IETF Logo Mail Archive

Re: [bmwg] Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03: (with COMMENT)

Adam Roach <adam@nostrum.com> Thu, 08 June 2017 13:32 UTC

Return-Path: <adam@nostrum.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CE5FA12EAF6; Thu, 8 Jun 2017 06:32:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.881
X-Spam-Level:
X-Spam-Status: No, score=-1.881 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RP_MATCHES_RCVD=-0.001, T_SPF_HELO_PERMERROR=0.01, T_SPF_PERMERROR=0.01] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WR9bKRos7KOY; Thu, 8 Jun 2017 06:32:32 -0700 (PDT)
Received: from nostrum.com (raven-v6.nostrum.com [IPv6:2001:470:d:1130::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2866212EAED; Thu, 8 Jun 2017 06:32:32 -0700 (PDT)
Received: from [172.18.0.15] (99-152-146-228.lightspeed.dllstx.sbcglobal.net [99.152.146.228]) (authenticated bits=0) by nostrum.com (8.15.2/8.15.2) with ESMTPSA id v58DWPhm097923 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 8 Jun 2017 08:32:26 -0500 (CDT) (envelope-from adam@nostrum.com)
X-Authentication-Warning: raven.nostrum.com: Host 99-152-146-228.lightspeed.dllstx.sbcglobal.net [99.152.146.228] claimed to be [172.18.0.15]
References: <149687493553.25630.10964914484670596603.idtracker@ietfa.amsl.com> <4D7F4AD313D3FC43A053B309F97543CF25FD6E3C@njmtexg5.research.att.com>
In-Reply-To: <4D7F4AD313D3FC43A053B309F97543CF25FD6E3C@njmtexg5.research.att.com>
Mime-Version: 1.0 (1.0)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="us-ascii"
Message-Id: <6EDBF7CC-F7E5-4437-96A9-5CB91A80A24A@nostrum.com>
Cc: The IESG <iesg@ietf.org>, "draft-ietf-bmwg-vswitch-opnfv@ietf.org" <draft-ietf-bmwg-vswitch-opnfv@ietf.org>, Sarah Banks <sbanks@encrypted.net>, "bmwg-chairs@ietf.org" <bmwg-chairs@ietf.org>, "bmwg@ietf.org" <bmwg@ietf.org>
X-Mailer: iPhone Mail (14E304)
From: Adam Roach <adam@nostrum.com>
Date: Thu, 08 Jun 2017 08:32:19 -0500
To: "MORTON, ALFRED C (AL)" <acmorton@att.com>
Archived-At: <https://mailarchive.ietf.org/arch/msg/bmwg/zc_RV7UqBmACXYQ-Jy8ea8O5uY8>
Subject: Re: [bmwg] Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03: (with COMMENT)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bmwg/>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 08 Jun 2017 13:32:34 -0000

Thanks. Your proposed clarification looks good to me. 

/a

> On Jun 7, 2017, at 18:20, MORTON, ALFRED C (AL) <acmorton@att.com> wrote:
> 
> Hi Adam,
> Thanks for your review, please see reply below.
> Al (for the co-authors)
> 
>> -----Original Message-----
>> From: Adam Roach [mailto:adam@nostrum.com]
>> Sent: Wednesday, June 07, 2017 6:36 PM
>> To: The IESG
>> Cc: draft-ietf-bmwg-vswitch-opnfv@ietf.org; Sarah Banks; bmwg-
>> chairs@ietf.org; sbanks@encrypted.net; bmwg@ietf.org
>> Subject: Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03:
>> (with COMMENT)
>> 
>> Adam Roach has entered the following ballot position for
>> draft-ietf-bmwg-vswitch-opnfv-03: No Objection
>> 
> ...
>> 
>> ----------------------------------------------------------------------
>> COMMENT:
>> ----------------------------------------------------------------------
>> 
>> I think this is a variation on the issue that Alissa calls out, but I'm
>> having a hard time reconciling:
>> 
>>   It's unlikely that the virtual switch will be the only application
>>   running on the System Under Test (SUT), so CPU utilization, Cache
>>   utilization, and Memory footprint should also be recorded for the
>>   virtual implementations of internetworking functions.
>> 
>> ...with...
>> 
>>   Further, benchmarking is performed on a "black-box" basis, relying
>>   solely on measurements observable external to the DUT/SUT.
>> 
>> Please add text that clarifies how the metrics that 3.1 says should be recorded
>> in section 3.1 relate to benchmarking.
>> 
> [ACM] 
> Ok, this is a somewhat different issue.
> There are metrics like CPU, Cache, and Memory Utilization
> which are reported by measurement processes running on the 
> Device Under Test. These are useful for the user/operations community
> to know, and may also be useful if there is a problem encountered 
> during testing. However, these metrics cannot be promoted to
> "Benchmark" status, because we cannot measure any of them 
> external to the DUT or SUT.  In fact, only the most important 
> externally observable metrics are designated as Benchmarks
> in our consensus process, and BMWG refers to internally measured
> metrics as "white-box metrics" to clearly distinguish them
> from the others.
> 
> So, I suggest adding the new sentence below:
>>   It's unlikely that the virtual switch will be the only application
>>   running on the System Under Test (SUT), so CPU utilization, Cache
>>   utilization, and Memory footprint should also be recorded for the
>>   virtual implementations of internetworking functions.
> +    However, internally-measured metrics such as these are not
> +    benchmarks; they should be useful for the audience (operations)
> +    to know, and may also be useful if there is a problem encountered 
> +    during testing.
> 
>

v2.23.0 | Report a Bug | By Email