Re: [bmwg] Benoit Claise's No Objection on draft-ietf-bmwg-vswitch-opnfv-03: (with COMMENT)

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Benoit,
Thanks for your review. please see replies below.
I really appreciate the specific pointers you've
provided for revisions to make this a more 
appropriate and valuable RFC.

Al (for the co-authors)

> -----Original Message-----
> From: Benoit Claise [mailto:[e-mail]]
> Sent: Thursday, June 08, 2017 8:09 AM
> To: The IESG
> Cc: [e-mail]; Sarah Banks; bmwg-
> [e-mail]; [e-mail]; [e-mail]; [e-mail]
> Subject: Benoit Claise's No Objection on draft-ietf-bmwg-vswitch-opnfv-
> 03: (with COMMENT)
> 
> Benoit Claise has entered the following ballot position for
> draft-ietf-bmwg-vswitch-opnfv-03: No Objection
...
> 
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> For sure, the topic of benchmarking of benchmarking virtual switches is
> important in the industry today.
> 
> Technical Summary: This draft summarizes the progress of the OPNFV project on
> virtual switch performance characterization, and notes that the OPNFV work
> builds upon the IETF work, particularly the BMWG work within IETF, with an
> emphasis on RFC2544.
> 
> I like the cross SDO aspects of this draft, as mentioned in the shepherd
> writeup: The document itself was reviewed in ETSI NFV, and used as a reference
> by at least one of the TST WG and IFA WF specifications (2 references in
> total). The document has also been reviewed by several individuals from the
> OPNFV community, and NFVRG (IRTF) is aware of the draft.
> 
> I would have preferred to see a document that specifies "benchmarking virtual
> switches" instead of "benchmarking virtual switches in OPNFV". In other words,
> just looking at the abstract, this is completely different to write:
> 
>    This memo describes the way to benchmark virtual switch performance.
>    The specifications are based on the Open Platform for NFV (OPNFV) project on
>    virtual switch performance "VSPERF".
> 
> As opposed to:
> 
>    This memo describes the progress of the Open Platform for NFV (OPNFV)
>    project on virtual switch performance "VSPERF".
[ACM] 
Yes.  We should have made some wording changes when this draft
transitioned from the Individual draft to WG item.
It would be great to be able to write your sentence, 
but the WG is not ready to do that.

However, we could say this instead:

This memo describes the contributions of the Open Platform 
for NFV (OPNFV) project on virtual switch performance "VSPERF", 
particularly in the areas of test set-ups and configuration 
parameters for the system under test.

and make some other updates, such as:

> 
> Regarding the "describes the progress of", sentences such as ...
>     - This project intends to build ...
[ACM] 
This project has extended ...

>     - Therefore, this memo begins to describe the additional
> considerations ...
[ACM] 
Therefore, this memo describes the additional

>     - This memo summarizes the progress of the Open Platform for NFV (OPNFV)
[ACM] 
This memo describes the contributions of the Open Platform...

>     project on virtual switch performance characterization, "VSPERF", through
>     the Brahmaputra (second) release [BrahRel].
[ACM] 
... through the Danube 3.0 (fourth) release to the chartered work 
of the BMWG (with stable references to their test descriptions). 

> ... lead me to think that this is unfinished work, or that work will anyway
> have to be updated.
[ACM] 
As I've mentioned in other comments, the initial rush of
test development has transitioned to inclusion of more
test devices, including open source testing software.

But I think with the emphasis on the specific contributions
and stable references, this memo is taking a more solid stand
(as much as anything is in our world where what we know changes fast).
 
> 
> Again, looking at the scope:
> 
>    The primary purpose and scope of the memo is to inform the industry
>    of work-in-progress that builds on the body of extensive BMWG
>    literature and experience, and describe the extensions needed for
>    benchmarking virtual switches.
> 
> I was hoping that the primary purpose and scope of the memo would be the
> specifications of the virtual switch benchmarking, not to inform about the
> work-in-progress.
[ACM] 
Yes, so let's say this instead:

The primary purpose and scope of the memo is to describe key 
aspects of vswitch benchmarking, particularly in the areas of test 
set-ups and configuration parameters for the system under test, 
and build on the body of extensive BMWG literature and experience.

> 
> Reading further, I see text such as:
> 
>     Specifications to be included in future updates of the LTD include:
> 
>        o  [RFC3918] Methodology for IP Multicast Benchmarking
> 
>        o  [RFC4737] Packet Reordering Metrics
> 
[ACM] turning this into a recommendation:
It is recommended that these references are included in future benchmarking specifications:

> or
> 
>    When considering characteristics important to "telco" network
>    functions, we must begin to consider additional performance metrics.
[ACM] 
When considering characteristics important to "telco" network 
functions, additional performance metrics are needed.
> 
> or
> 
>    Tests have been (or will be) designed to collect the metrics below:
[ACM] (these are all included now):
Tests have been designed to collect the metrics below:

> 
> On one side, I appreciate that you're publishing what you learned, which should
> lead up to revisions in the future. On the other side, I've not been used to
> that approach in BMWG. I'll trust the responsible AD regarding the publication
> of this document as a kind of intermediary document.
[ACM] 
Thank you. We know we have a lot more to learn before we 
update truly fundamental RFCs like 2544 and 2889.
That would be a big step, so we need to be very sure.

> 
> I believe the document would be a better flow (and that some concerns would
> disappear) if it would have the following structure
>     - benchmarking virtual switches is key in the industry today
>     - there are the virtual switches benchmarking specification today
>     - we worked on this with OPNFV (VSPERF). Great collaboration.
>     - this is a complex and evolving field:
>       compared to RFC 2544, we learn that ...
>       we realize that this is work in progress ...
>     - future plan: we will need to integrate more "stuff" in the benchmark,
>     such as RFC3918, RFC4737, additional perf metrics, you-name-it - we'll work
>     with OPNFV (and other groups) on this future plan.
[ACM] 
Hopefully, the suggested changes above will accomplish that,
but some of your ideas can be easily added to the Introduction.
Please trust me for that text...
> 
> Editorial
> - Section 3.4. Flow timeout.
> You mean the active timeout, section 2.2.23 in RFC 6645?
> Should this section refer to RFC6645, or RFC 5470 (section 5.1.1)?
> 
[ACM] 
Ben raised a similar comment.  
I'll add the best reference for BMWG docs, which is

For example, RFC 2889 says:
   " ... The DUT/SUT address aging time SHOULD be configured to be greater than
   the period of the learning phase of the test plus the trial duration
   plus any configuration time required by the testing device."
See:
https://tools.ietf.org/html/rfc2889#section-3
The learning phase is benchmarked separately.


> -
> 
>    In addition to this, the LTD also re-uses the terminology defined by:
> 
>    o  [RFC2285] Benchmarking Terminology for LAN Switching Devices
> 
>    o  [RFC5481] Packet Delay Variation Applicability Statement
> 
> RFC5481 is already mentioned in the previous bullet list (The base performance
> tests in the LTD are based on the pre-existing specifications developed by BMWG
> to test the performance of physical switches. These specifications include). So
> hopefully, it re-uses the terminology, right?
[ACM] 
Right, 5481 removed.

Except for the Intro paragraph I mentioned, 
the working text is revised as described.
>