Re: [bmwg] WGLC: draft-ietf-bmwg-ipv6-tran-tech-benchmarking-02

["MORTON, ALFRED C (AL)" <[e-mail]>]

Hi Marius,

Here are my comments on your draft (beyond the observation
that this is an excellent piece of work, I've said that many times...)
In these comments, I marked the margin with "|" where it might be 
difficult to find the text I suggest to change.

regards,
Al
(as a participant)

Section 1., paragraph 4:
OLD:

    The document also includes an approach to quantify load scalability.
    Load scalability can be defined as a system's ability to gracefully
    accommodate higher loads. Because poor scalability usually leads to
    poor performance, the proposed approach is to quantify the load
    scalability by measuring the performance degradation created by a
    higher number of network flows.

NEW:

    The document also includes an approach to quantify load scalability.
    Load scalability can be defined as a system's ability to gracefully
    accommodate higher loads. Because poor scalability usually leads to
    poor performance, the proposed approach is to quantify the load
    scalability by measuring the performance degradation created by a
    higher number of network flows.
 [ACM] maybe this instead: (?)
    This document also includes an approach to quantify flow scalability.
    Flow scalability can be defined as a system's ability to gracefully
    accommodate increasing numbers of flows.
    The approach taken here is to quantify the flow scalability
    by measuring the performance created by a number (>>1) of
    network flows, and comparing performance to the single flow case.
 
 [ACM] How hard would it be to add this???
    The document also includes an approach to quantify performance
    when operating in overload. Overload scalability can be defined as
    a system's ability to gracefully accommodate greater numbers of flows
    than the maximum number of flows which the DUT can operate normally.
    The approach taken here is to quantify the Overload scalability
    by measuring the performance created by an excessive number of
    network flows, and comparing performance to the non-overload case.


Section 1.1., paragraph 6:
OLD:

    Note: X,Y are part of the {4,6} set.

NEW:

 |  Note: X,Y are part of the set {4,6}, and X NOT.EQUAL Y.


Section 3., paragraph 0:
OLD:

    3. Encapsulation: The production network is assumed to have all
       three domains, Domains A and B are IPvX specific, while the core
       domain is IPvY specific. An encapsulation mechanism is used to
       traverse the core domain. The IPvX packets are encapsulated to
       IPvY packets at the edge between Domain A and the Core domain.
       Subsequently, the IPvY packets are decapsulated at the edge
       between the Core domain and Domain B.

NEW:

    3. Encapsulation: The production network is assumed to have all
       three domains, Domains A and B are IPvX specific, while the core
       domain is IPvY specific. An encapsulation mechanism is used to
       traverse the core domain. The IPvX packets are encapsulated to
       IPvY packets at the edge between Domain A and the Core domain.
 |     Subsequently, the IPvY packets are de-encapsulated at the edge
       between the Core domain and Domain B.


Section 2., paragraph 3:
OLD:

    Although these terms are usually associated with protocol
    requirements, in this doc the terms are requirements for users and
    systems that intend to implement the test conditions and claim
    conformance with this specification.

NEW:

    Although these terms are usually associated with protocol
 |  requirements, in this document the terms are requirements for users and
    systems that intend to implement the test conditions and claim
    conformance with this specification.


Section 4.2., paragraph 1:
OLD:

    For evaluating the performance of Encapsulation and Double
    translation transition technologies, a dual DUT setup (see Figure 2)
    SHOULD be employed. The tester creates a network flow of IPvX
    packets. The first DUT is responsible for the encapsulation or
    translation of IPvX packets into IPvY packets. The IPvY packets are
    decapsulated/translated back to IPvX packets by the second DUT and
    forwarded to the tester.

NEW:

    For evaluating the performance of Encapsulation and Double
    translation transition technologies, a dual DUT setup (see Figure 2)
    SHOULD be employed. The tester creates a network flow of IPvX
    packets. The first DUT is responsible for the encapsulation or
    translation of IPvX packets into IPvY packets. The IPvY packets are
 |  de-encapsulated/translated back to IPvX packets by the second DUT and
    forwarded to the tester.


Section 4.2., paragraph 4:
OLD:

    Note: For encapsulation IPv6 transition technologies, in the single
    DUT setup, in order to test the decapsulation efficiency, the tester
    SHOULD be able to send IPvX packets encasulated as IPvY.

NEW:

    Note: For encapsulation IPv6 transition technologies, in the single
 |  DUT setup, in order to test the de-encapsulation efficiency, the tester
 |  SHOULD be able to send IPvX packets encapsulated as IPvY.


Section 6., paragraph 1:
OLD:

    The idea of testing under different operational conditions was first
    introduced in [RFC2544](Section 11) and represents an important
    aspect of benchmarking network elements, as it emulates to some
    extent the conditions of a production environment. [RFC5180]
    describes complementary testing conditions specific to IPv6. Their
    recommendations can be referred for IPv6 transition technologies
    testing as well.

NEW:

    The idea of testing under different operational conditions was first
    introduced in [RFC2544](Section 11) and represents an important
    aspect of benchmarking network elements, as it emulates to some
 |  extent the conditions of a production environment. Section ?? of [RFC5180]
    describes complementary testing conditions specific to IPv6. Their
    recommendations can be referred for IPv6 transition technologies
    testing as well.


Section 7., paragraph 2:
OLD:

 7.1. Throughput - [RFC2544]

NEW:

 7.1. Throughput
 
 |  Use Section ?? of [RFC2544] unmodified.


Section 7.2., paragraph 12:
OLD:

       The test MUST be repeated at least 20 times with the reported
    value being the median of the recorded values.

NEW:

 |  The Latency test MUST be repeated at least 20 times with the reported
 |  value being the median of the recorded values for TL and WCL (??).


Section 7.3.2., paragraph 6:
OLD:

    Where: Dmin - the minimum One-way delay in the stream

NEW:

 |   Where: Dmin - the minimum One-way IPDV in the stream


Section 7.3.2., paragraph 7:
OLD:

           Dmed - the median One-way delay of the stream

NEW:

 |          Dmed - the median One-way IPDV of the stream


Section 7.3.2., paragraph 8:
OLD:

           Dmax - the maximum One-way delay in the stream

NEW:

 |          Dmax - the maximum One-way IPDV in the stream


Section 7.3.2., paragraph 11:
OLD:

 7.4. Frame Loss Rate - [RFC2544]

NEW:

 7.4. Frame Loss Rate


Section 7.3.2., paragraph 12:
OLD:

 7.5. Back-to-back Frames - [RFC2544]

NEW:

 |  Use Section ?? of [RFC2544] unmodified.


Section 7.3.2., paragraph 13:
OLD:

 7.6. System Recovery - [RFC2544]

NEW:

 7.5. Back-to-back Frames


Section 7.3.2., paragraph 14:
OLD:

 7.7. Reset - [RFC2544]

NEW:

 |  Use Section ?? of [RFC2544] unmodified.
 
 7.6. System Recovery
 
 |  Use Section ?? of [RFC2544] unmodified.
 
 7.7. Reset
 
 |  Use Section ?? of [RFC6201] unmodified.


Section 8., paragraph 3:
OLD:

 8.1. Concurrent TCP Connection Capacity -[RFC3511]

NEW:

 8.1. Concurrent TCP Connection Capacity


Section 8., paragraph 4:
OLD:

 8.2. Maximum TCP Connection Establishment Rate -[RFC3511]

NEW:

 |  Use Section ?? of [3511] unmodified.
 
 8.2. Maximum TCP Connection Establishment Rate
 
 |  Use Section ?? of [3511] unmodified.


Section 60, paragraph 0:
OLD:

    Procedure: Send a specific number of DNS queries at a specific rate
    to the DUT and then count the replies received in time (within a
    predefined timeout period from the sending time of the corresponding
    query, having the default value 1 second) from the DUT. If the count
    of sent queries is equal to the count of received replies, the rate
    of the queries is raised and the test is rerun. If fewer replies are
    received than queries were sent, the rate of the queries is reduced
    and the test is rerun. The duration of the test SHOULD be at least
    60 seconds to reduce the potential gain of a DNS64 server, which is
    able to exhibit higher performance by storing the requests and thus
    utilizing also the timeout time for answering them. For the same
    reason, no higher timeout time than 1 second SHOULD be used.

NEW:

    Procedure: Send a specific number of DNS queries at a specific rate
    to the DUT and then count the replies received in time (within a
    predefined timeout period from the sending time of the corresponding
    query, having the default value 1 second) from the DUT. If the count
    of sent queries is equal to the count of received replies, the rate
    of the queries is raised and the test is rerun. If fewer replies are
    received than queries were sent, the rate of the queries is reduced
 |  and the test is rerun. The duration of the each trial SHOULD be at least
    60 seconds to reduce the potential gain of a DNS64 server, which is
    able to exhibit higher performance by storing the requests and thus
    utilizing also the timeout time for answering them. For the same
    reason, no higher timeout time than 1 second SHOULD be used.


Section 60, paragraph 1:
OLD:

    The number of processed DNS queries per second is the fastest rate
    at which the count of DNS replies sent by the DUT is equal to the
    number of DNS queries sent to it by the test equipment.

NEW:

 |  The maximum number of processed DNS queries per second is the fastest rate
    at which the count of DNS replies sent by the DUT is equal to the
    number of DNS queries sent to it by the test equipment.


Section 10., paragraph 0:
OLD:

 10. Scalability

NEW:

 10. Scalability
 |  [ACM] If we agree on a terminology change, there are edits needed below.


Section 10.2.1., paragraph 2:
OLD:

    The same tests have to be repeated for n network flows, where the
    network flows are started simultaneously. The performance
    degradation of the X benchmarking dimension SHOULD be calculated as
    relative performance change between the 1-flow results and the n-
    flow results, using the following formula:

NEW:

    The same tests have to be repeated for n network flows, where the
    network flows are started simultaneously. The performance
    degradation of the X benchmarking dimension SHOULD be calculated as
 |  relative performance change between the 1-flow (single flow) results and the n-
    flow results, using the following formula:


Section 12., paragraph 1:
OLD:

    To ensure the stability of the benchmarking scores obtained using
    the tests presented in Sections 6-9, multiple test iterations are
    RECOMMENDED. Using a summarizing function (or measure of central
    tendency) can be a simple and effective way to compare the results
    obtained across different iterations. However, over-summarization is
    an unwanted effect of reporting a single number.

NEW:

    To ensure the stability of the benchmarking scores obtained using
 |  the tests presented in Sections 6 through 9, multiple test iterations are
    RECOMMENDED. Using a summarizing function (or measure of central
    tendency) can be a simple and effective way to compare the results
    obtained across different iterations. However, over-summarization is
    an unwanted effect of reporting a single number.


Section 12., paragraph 3:
OLD:

    To that end, data presented in [ietf95pres] indicate the median as
    suitable summarizing function and the 1st and 99th percentiles as
    variation measures for DNS Resolution Performance and PDV. . The
    median and percentile calculation functions SHOULD follow the
    recommendations of [RFC2330] Section 11.3.

NEW:

    To that end, data presented in [ietf95pres] indicate the median as
    suitable summarizing function and the 1st and 99th percentiles as
 |  variation measures for DNS Resolution Performance and PDV. The
    median and percentile calculation functions SHOULD follow the
    recommendations of [RFC2330] Section 11.3.


Section 14., paragraph 1:
OLD:

    The IANA has allocated the prefix 2001:0002::/48 [RFC5180] for IPv6
    benchmarking. For IPv4 benchmarking, the 198.18.0.0/15 prefix was
    reserved, as described in [RFC6890]. The two ranges are sufficient
    for benchmarking IPv6 transition technologies.

NEW:

    The IANA has allocated the prefix 2001:0002::/48 [RFC5180] for IPv6
    benchmarking. For IPv4 benchmarking, the 198.18.0.0/15 prefix was
    reserved, as described in [RFC6890]. The two ranges are sufficient
 |  for benchmarking IPv6 transition technologies. Thus, no action is requested.

> -----Original Message-----
> From: bmwg [mailto:[e-mail]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Monday, October 10, 2016 11:25 AM
> To: [e-mail]
> Subject: [bmwg] WGLC: draft-ietf-bmwg-ipv6-tran-tech-benchmarking-02
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> 
> BMWG:
> 
> A WG Last Call period for the Internet-Draft on
> Benchmarking Methodology for IPv6 Transition Technologies:
> 
> https://tools.ietf.org/html/draft-ietf-bmwg-ipv6-tran-tech-benchmarking-
> 02
> 
> will be open from 10 October 2016 through 24 October 2016**.
> 
> This is the first WGLC on this draft, following the BMWG Last Call
> Process.
> See
> http://www.ietf.org/mail-archive/web/bmwg/current/msg00846.html
> 
> Please read the draft and express your opinion on whether or not it
> should be forwarded to the Area Directors for publication as an
> Informational RFCs.  Send your comments to this list or to the
> co-chairs: [e-mail] and [e-mail]
> 
> for the co-chairs,
> Al
> 
> ** continuing comments are welcome through the IETF-97 meeting
> 
> _______________________________________________
> bmwg mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/bmwg