Re: [bmwg] Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03: (with DISCUSS and COMMENT)

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Alissa,
Thanks for your review, please see reply below.
Al (for the co-authors)

> -----Original Message-----
> From: Alissa Cooper [mailto:[e-mail]]
> Sent: Wednesday, June 07, 2017 5:53 PM
> To: The IESG
> Cc: [e-mail]; Sarah Banks; bmwg-
> [e-mail]; [e-mail]; [e-mail]
> Subject: Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03:
> (with DISCUSS and COMMENT)
...
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> Picking up on the thread that started with the Gen-ART review, I'm not clear on
> the position of this document when it comes to repeatability. If all of the
> parameters listed in Section 3.3 (assuming they apply) are configured and
> documented, is it assumed that the benchmarks will be repeatable and comparable
> with hardware implementation benchmarks?
[ACM] 
>>>
Of course, it has been the goal of the project to produce 
repeatable results, and a large set of the parameters
believed to be critical was provided so that the benchmarking
community could better appreciate the increase in configuration
complexity inherent in this work. So, yes, this set was assumed 
sufficient for the infrastructure in use by the VSPERF project
to obtain repeatable results from test-to-test. 
>>> We could add summary above as a new paragraph in section 3.3.

Benchmark Comparability between virtual and physical/hardware 
implementations of equivalent functions will likely place more
detailed and exact requirements on the *testing systems*
(in terms of stream generation, algorithms to search 
for max values, and their configurations of course). 
This is another area for standardization to appreciate,
now that we have a few years testing experience.
However, the is a topic for a future draft.

> Or is the position of the document
> that the benchmarks are not likely to be repeatable/comparable in some (many?)
> cases, given the increase in complexity? Or that more work needs to be done
> (outside the specification process) to achieve repeatability?
[ACM] 
Thanks to some server re-arrangements in the lab that hosts
the VSPERF project, we are able to evaluate our long-term
result repeatability and repeatability across two similar
hardware platforms in different buildings. I'm still processing
some of the results we will present next week at the OPNFV Summit,
but the cross hardware platform results look quite consistent,
even with a benchmark where Scott Bradner experienced some
repeatability problems in his lab back in the 90’s.

> I think this needs to be more clear in the document.
[ACM] 
If the >>>paragraph<<< accomplishes that for you,
with some minor tweaks to word it less like a reply, let me know.

> 
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> I understand the debate about the publication of this document as an IETF
> consensus RFC. To me it seems valuable to publish and that this item is within
> the WG's charter. I think adding the updates necessary to make the document
> up-to-date with the Danube 3.0 release as suggested in the thread with Alvaro
> would be valuable.
> 
[ACM] 
Great, I've located the list of tests added in Colorado and Danube,
so I'll add them in section 5, and update the release info accordingly.