Re: [bmwg] Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03: (with COMMENT)

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Adam,
Thanks for your review, please see reply below.
Al (for the co-authors)

> -----Original Message-----
> From: Adam Roach [mailto:[e-mail]]
> Sent: Wednesday, June 07, 2017 6:36 PM
> To: The IESG
> Cc: [e-mail]; Sarah Banks; bmwg-
> [e-mail]; [e-mail]; [e-mail]
> Subject: Adam Roach's No Objection on draft-ietf-bmwg-vswitch-opnfv-03:
> (with COMMENT)
> 
> Adam Roach has entered the following ballot position for
> draft-ietf-bmwg-vswitch-opnfv-03: No Objection
> 
...
>  
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> I think this is a variation on the issue that Alissa calls out, but I'm
> having a hard time reconciling:
> 
>    It's unlikely that the virtual switch will be the only application
>    running on the System Under Test (SUT), so CPU utilization, Cache
>    utilization, and Memory footprint should also be recorded for the
>    virtual implementations of internetworking functions.
> 
> ...with...
> 
>    Further, benchmarking is performed on a "black-box" basis, relying
>    solely on measurements observable external to the DUT/SUT.
> 
> Please add text that clarifies how the metrics that 3.1 says should be recorded
> in section 3.1 relate to benchmarking.
> 
[ACM] 
Ok, this is a somewhat different issue.
There are metrics like CPU, Cache, and Memory Utilization
which are reported by measurement processes running on the 
Device Under Test. These are useful for the user/operations community
to know, and may also be useful if there is a problem encountered 
during testing. However, these metrics cannot be promoted to
"Benchmark" status, because we cannot measure any of them 
external to the DUT or SUT.  In fact, only the most important 
externally observable metrics are designated as Benchmarks
in our consensus process, and BMWG refers to internally measured
metrics as "white-box metrics" to clearly distinguish them
from the others.

So, I suggest adding the new sentence below:
>    It's unlikely that the virtual switch will be the only application
>    running on the System Under Test (SUT), so CPU utilization, Cache
>    utilization, and Memory footprint should also be recorded for the
>    virtual implementations of internetworking functions.
+    However, internally-measured metrics such as these are not
+    benchmarks; they should be useful for the audience (operations)
+    to know, and may also be useful if there is a problem encountered 
+    during testing.