Re: [bmwg] Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03: (with DISCUSS and COMMENT)

Hi there,

The authors have just integrated these changes. I think that they
address your concern, and so I asked them to post a new version -
https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/

Please let me / them know if they correctly address your DISCUSS.
W

On Thu, Jun 8, 2017 at 7:36 AM, Alissa Cooper <alissa@cooperw.in> wrote:
> Hi Al,
>
>> On Jun 7, 2017, at 6:57 PM, MORTON, ALFRED C (AL) <acmorton@att.com> wrote:
>>
>> Hi Alissa,
>> Thanks for your review, please see reply below.
>> Al (for the co-authors)
>>
>>> -----Original Message-----
>>> From: Alissa Cooper [mailto:alissa@cooperw.in]
>>> Sent: Wednesday, June 07, 2017 5:53 PM
>>> To: The IESG
>>> Cc: draft-ietf-bmwg-vswitch-opnfv@ietf.org; Sarah Banks; bmwg-
>>> chairs@ietf.org; sbanks@encrypted.net; bmwg@ietf.org
>>> Subject: Alissa Cooper's Discuss on draft-ietf-bmwg-vswitch-opnfv-03:
>>> (with DISCUSS and COMMENT)
>> ...
>>>
>>> ----------------------------------------------------------------------
>>> DISCUSS:
>>> ----------------------------------------------------------------------
>>>
>>> Picking up on the thread that started with the Gen-ART review, I'm not clear on
>>> the position of this document when it comes to repeatability. If all of the
>>> parameters listed in Section 3.3 (assuming they apply) are configured and
>>> documented, is it assumed that the benchmarks will be repeatable and comparable
>>> with hardware implementation benchmarks?
>> [ACM]
>>>>>
>> Of course, it has been the goal of the project to produce
>> repeatable results, and a large set of the parameters
>> believed to be critical was provided so that the benchmarking
>> community could better appreciate the increase in configuration
>> complexity inherent in this work. So, yes, this set was assumed
>> sufficient for the infrastructure in use by the VSPERF project
>> to obtain repeatable results from test-to-test.
>>>>> We could add summary above as a new paragraph in section 3.3.
>>
>> Benchmark Comparability between virtual and physical/hardware
>> implementations of equivalent functions will likely place more
>> detailed and exact requirements on the *testing systems*
>> (in terms of stream generation, algorithms to search
>> for max values, and their configurations of course).
>> This is another area for standardization to appreciate,
>> now that we have a few years testing experience.
>> However, the is a topic for a future draft.
>>
>>> Or is the position of the document
>>> that the benchmarks are not likely to be repeatable/comparable in some (many?)
>>> cases, given the increase in complexity? Or that more work needs to be done
>>> (outside the specification process) to achieve repeatability?
>> [ACM]
>> Thanks to some server re-arrangements in the lab that hosts
>> the VSPERF project, we are able to evaluate our long-term
>> result repeatability and repeatability across two similar
>> hardware platforms in different buildings. I'm still processing
>> some of the results we will present next week at the OPNFV Summit,
>> but the cross hardware platform results look quite consistent,
>> even with a benchmark where Scott Bradner experienced some
>> repeatability problems in his lab back in the 90’s.
>>
>>> I think this needs to be more clear in the document.
>> [ACM]
>> If the >>>paragraph<<< accomplishes that for you,
>> with some minor tweaks to word it less like a reply, let me know.
>
> I think both of your first two paragraphs above make useful points for inclusion in the draft, so if you’re able to write up some text that incorporates them, that would resolve my issue.
>
> Thanks,
> Alissa
>
>
>>
>>>
>>>
>>> ----------------------------------------------------------------------
>>> COMMENT:
>>> ----------------------------------------------------------------------
>>>
>>> I understand the debate about the publication of this document as an IETF
>>> consensus RFC. To me it seems valuable to publish and that this item is within
>>> the WG's charter. I think adding the updates necessary to make the document
>>> up-to-date with the Danube 3.0 release as suggested in the thread with Alvaro
>>> would be valuable.
>>>
>> [ACM]
>> Great, I've located the list of tests added in Colorado and Danube,
>> so I'll add them in section 5, and update the release info accordingly.
>>
>>
>
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://www.ietf.org/mailman/listinfo/bmwg

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf