[bmwg] Comments on draft-ietf-bmwg-ca-bench-meth-01

[Al Morton <[e-mail]>]

Mike and Sarah,

Comments below, some are purely editorial.

Al
(as a participant)

-=-=-=-=-=-=-

Section 1 Intro
...
    The end goal of this methodology is to generate performance metrics
    in a lab environment that will more closely relate to actual observed
s/more closely/closely/

    performance on production networks.  By utilizing dynamic traffic
    patterns relevant to modern networks, this methodology should be able
    to more closely tie laboratory and production metrics.
s/more closely/closely/

and a question about:
   ... It should be
    further noted than any metrics acquired from production networks
    SHOULD be captured according to the policies and procedures of the
    IPPM or PMOL working groups.

What metrics are you thinking about here?

-=-=-=-=-=-=-

Section 3.2

    ...Application Flows,
    as defined in RFC 2722 [9] are able to be well-defined without simply
    referring to a network capture.  An example traffic template is
    defined and listed in Appendix A of this document.

Actually, RFC 2722 defines a Traffic Flow, not application:

3 Traffic Flows and Reporting Granularity

    A flow was defined in section 2.1 above in abstract terms as follows:

        "A TRAFFIC FLOW is an artifical logical equivalent to a call or
        connection, belonging to a (user-specieied) METERED TRAFFIC
        GROUP."

-=-=-=-=-=-=-=-

Section 3.4
...
    If we change the example slightly and increase the size of each
    datagram to 1312 bytes, then it becomes necessary to recompute the
    math.
s/math/load/

-=-=-=-=-=-=-=-

3.7.3.  TCP Stack Considerations

    The IETF has historically provided guidance and information on TCP
    stack considerations.  This methodology is strictly focused on
    performance metrics at layers above 4, thus does not specifically
    define any TCP stack configuration parameters of either the tester or
    the DUTs.  The TCP configuration of the tester MUST remain constant
    across all DUTs in order to ensure comparable results.  While the
    following list of references is not exhaustive, each document
    contains a relevant discussion on TCP stack considerations.

    Congestion control algorithms are discussed in Section 2 of RFC 3148
    [11] with even more detailed references.  TCP receive and congestion
    window sizes are discussed in detail in RFC 6349 [12].

It would be useful to cite the TCP Raodmap in the first paragraph above:
http://tools.ietf.org/html/rfc4614


-=-=-=-=-=-=-=-=-=-

4.1.3 Procedure

It's worth noting that when the offered load results in
100% link utilization on all ingress links,
then the DUT has achieved "wire-speed performance",
or words to that effect.

...
    This procedure MAY be repeated any number of times with the results
    being averaged together.
s/number/reasonable number/

-=-=-=-=-=-=-=-=-=-

4.1.4.3.  Packet Loss

    The test tool SHOULD report the number of flow packets lost or
    dropped from source to destination.
s/flow packets/packets/ (i think)

-=-=-=-=-=-=-=-=-=-

4.2.4.5.  Packet Loss

    The test tool SHOULD report the number of flow packets lost or
    dropped from source to destination.

Remove - redundant with 4.2.4.2

-=-=-=-=-=-=-=-=-=-

4.2.4.6.  Application Flow Latency

    The test tool SHOULD report the minimum, maximum and average amount
    of time an application flow member takes to traverse the DUT, as
    defined by RFC 1242 [3], Section 3.13.

What is a "flow member"?
Are these statistics calculated over all packets in a flow?

    ...                                    This rate SHOULD be reported
    individually for each application protocol present within the traffic
    mix.

Not a rate, latency is in units of seconds.

-=-=-=-=-=-=-=-=-=-

Appendix B

Good!