Re: [bmwg] AD review of draft-ietf-bmwg-dcbench-terminology.

[Lucien Avramov <[e-mail]>]

Thanks Warren!!

Submited -09 with all the modifications!

Cheers,
Lucien

On Tue, May 23, 2017 at 11:53 AM, Warren Kumari <[e-mail]> wrote:

> Hi there,
>
> Apologies, I had forgotten to click the AD eval button... {DONE}
>
> Thanks for a (generally) clear and useful document.
>
> I have completed my AD review of draft-ietf-bmwg-dcbench-terminology and
> I have a number of suggested edits to make the document even more readable
> / clearer. In a few instances I wasn't quite sure what the document was
> trying to say, so please let me know if any of the suggestions don't make
> sense, or change the text meaning...
>
> Also, please poke me once updated, so that I can progress it to IETF LC.
>
> W
> ------
>
> Abstract
>
> The purpose of this informational document is to establish definitions,
> discussion and measurement techniques for data center benchmarking.
> [O] establish definitions, discussion and measurement techniques
> [R] I'm not sure what is meant by "discussion" here. "Discussion and
> measurement techniques" doesn't make much sense to me. Perhaps "to
> establish definitions, foster discussion and describe measurement
> techniques ..."?
>
>
>
> Also, it is to introduce new terminologies applicable to data center
> [O] benchmarking. Also, it is to introduce
> [P] benchmarking, as well as introduce
> [R] clarity/no need for a separate sentence.
>
> performance evaluations. The purpose of this document is not to define
> the test methodology, but rather establish the important concepts when
> one is interested in benchmarking network switches and routers in the
> [O] when one is interested in benchmarking
> [P] for benchmarking
> [R] readability
>
> data center.
>
>
>
> 1.  Introduction
>
>    Traffic patterns in the data center are not uniform and are contently
> [O] contently
> [P] constantly
> [R] word choice/typo
>
>    changing. They are dictated by the nature and variety of applications
>    utilized in the data center. It can be largely east-west traffic
>    flows in one data center and north-south in another, while some may
>    combine both. Traffic patterns can be bursty in nature and contain
>    many-to-one, many-to-many, or one-to-many flows. Each flow may also
>    be small and latency sensitive or large and throughput sensitive
>    while containing a mix of UDP and TCP traffic. All of which can
>    coexist in a single cluster and flow through a single network device
>    all at the same time. Benchmarking of network devices have long used
> [O] All of which can
>    coexist in a single cluster and flow through a single network device
>    all at the same time.
> [P] One or more of these may coexist in a single cluster and flow through
> a single network device simultaneously.
> [R] grammar/readability - unless this changes the intent?
>
>    [RFC1242], [RFC2432], [RFC2544], [2] and [3]. These benchmarks have
>    largely been focused around various latency attributes and max
>    throughput of the Device Under Test being benchmarked. These
>    standards are good at measuring theoretical max throughput,
>    forwarding rates and latency under testing conditions, but to not
> [O] , but to not
> [P]; but they do not
> [R] grammar
>
>    represent real traffic patterns that may affect these networking
>    devices. The data center networking devices covered are switches and
>    routers.
>
>
>
> 1.2. Definition format
>
>    Term to be defined. (e.g., Latency)
>
>    Definition: The specific definition for the term.
>
>    Discussion: A brief discussion about the term, it's application and
> [O] it's
> [P] its
> [R] Grammar. Should be a possessive, not a contraction of "it is."
>    any restrictions on measurement procedures.
>
>    Measurement Units: Methodology for the measure and units used to
>    report measurements of this term, if applicable.
>
>
> 2.  Latency
>
>
> 2.1. Definition
>
>    Latency is a the amount of time it takes a frame to transit the DUT.
>    Latency is measured in unit of time (seconds, milliseconds,
>    microseconds and so on). The purpose of measuring latency is to
>    understand what is the impact of adding a device in the communication
> [O]  understand what is the impact
> [P] understand the impact
> [R] grammar
>
>    path.
>
>    The Latency interval can be assessed between different combinations
>    of events, irrespectively of the type of switching device (bit
> [O] irrespectively
> [P] regardless
> [R] grammar
>
>    forwarding aka cut-through or store forward type of device)
>
>
>
> 2.2 Discussion
>
>    FILO is the most important measuring definition. Any type of switches
> [O] Any type of switches
> [P] All switches
> [R] readability
>
>    MUST be measured with the FILO mechanism: FILO will include the
>    latency of the switch and the latency of the frame as well as the
>    serialization delay. It is a picture of the 'whole' latency going
>    through the DUT. For applications, which are latency sensitive and
>    can function with initial bytes of the frame, FIFO MAY be an
>    additional type of measuring to supplement FILO.
>
>
> 3 Jitter
> 3.2 Discussion
>
>    In addition to PDV Range and or a high percentile of PDV, Inter-
> [O]  and or
> [P] and/or
>
>    Packet Delay Variation (IPDV) as defined in section 4.1 of RFC5481
>    (differences between two consecutive packets) MAY be used for the
>    purpose of determining how packet spacing has changed during
>    transfer, for example to see if packet stream has become closely-
> [O] for example to see
> [P] for example, to see
> [R] grammar
>
>    spaced or "bursty". However, the Absolute Value of IPDV SHOULD NOT be
>    used, as this collapses the "bursty" and "dispersed" sides of the
>    IPDV distribution together.
>
> 4 Physical Layer Calibration
> 4.2 Discussion
>
>    Physical layer calibration is part of the end to end latency, which
>    should be taken into acknowledgment while evaluating the DUT. Small
>    variations of the physical components of the test may impact the
>    latency being measure so they MUST be described when presenting
> [O] latency being measure so
> [P] latency being measured, so
> [R] grammar
>
>    results.
>
>
> 5 Line rate
>
> 5.1 Definition
>
>    The transmit timing, or maximum transmitted data rate is controlled
>    by the "transmit clock" in the DUT.  The receive timing (maximum
>    ingress data rate) is derived from the transmit clock of the
>    connected interface.
>
>    The line rate or physical layer frame rate is the maximum capacity to
>    send frames of a specific size at the transmit clock frequency of the
>    DUT.
>
>    The term port capacity term defines the maximum speed capability for
> [O] The term port capacity term
> [P] The term "port capacity"
> [R] readability
>
>    the given port; for example 1GE, 10GE, 40GE, 100GE etc.
>
>    The frequency ("clock rate") of the transmit clock in any two
>    connected interfaces will never be precisely the same, therefore a
>    tolerance is needed, this will be expressed by Parts Per Million
> [O] same, therefore a tolerance is needed, this will
> [P] same; therefore, a tolerance is needed. This will
> [R] grammar
>
>
> 5.2 Discussion
>
>    For a transmit clock source, most Ethernet switches use "clock
>    modules" (also called "oscillator modules") that are sealed,
>    internally temperature-compensated, and very accurate. The output
>    frequency of these modules is not adjustable because it is not
>    necessary.  Many test sets, however, offer a software-controlled
>    adjustment of the transmit clock rate, which should be used to
>    compensate the test equipment to not send more than line rate of the
>    DUT.
>
> [O] which should be used to
>    compensate the test equipment to not send more than line rate of the
>    DUT.
> [R] please reword for clarity.
>
>    To allow for the minor variations typically found in the clock rate
>    of commercially-available clock modules and other crystal-based
>
>
>    Test set equipment manufacturers are well-aware of the standards, and
>    allows a software-controlled +/- 100 PPM "offset" (clock-rate
> [O] allows
> [P] allow
> [R] grammar
>
>    adjustment) to compensate for normal variations in the clock speed of
>    "devices under test". This offset adjustment allows engineers to
>    determine the approximate speed the connected device is operating,
>    and verify that it is within parameters allowed by standards.
>
>
>
> 5.3 Measurement Units
>
>
>    In a production network, it is very unlikely to see precise line rate
>    over a very brief period. There is no observable difference between
>    dropping packets at 99% of line rate and 100% of line rate. -Line
>    rate CAN measured at 100% of line rate with a -100PPM adjustment. -
>    Line rate SHOULD be measured at 99,98% with 0 PPM adjustment.-The PPM
> [O] .-The
> [P] . The
> [R] Typo Or was this intended to be a list?
>
>    adjustment SHOULD only be used for a line rate type of measurement
>
>
> 6  Buffering
> 6.1 Buffer
> 6.1.1 Definition
>
>    Buffer Size: the term buffer size, represents the total amount of
> [O] the term buffer size, represents
> [P] the term buffer size represents
> [R] grammar
>
>    frame buffering memory available on a DUT. This size is expressed in
>    Byte; KB (kilobytes), MB (megabytes) or GB (gigabyte). When the
>    buffer size is expressed it SHOULD be defined by a size metric
>    defined above. When the buffer size is expressed, an indication of
> [O]  defined above
> [P] listed above [OR] stated above
> [R] the sizes are not "defined" above
>
>    the frame MTU used for that measurement is also necessary as well as
>    the cos or dscp value set; as often times the buffers are carved by
>    quality of service implementation. (please refer to the buffer
> [O] . (please
> [P] . (Please
> [R] grammar
>
>    efficiency section for further details).
>
>    Example: Buffer Size of DUT when sending 1518 bytes frames is 18 Mb.
>
>    Port Buffer Size: the port buffer size is the amount of buffer a
>    single ingress port, egress port or combination of ingress and egress
>    buffering location for a single port. The reason of mentioning the
> [O] reason of mentioning
> [P] reason for mentioning
> [R] grammar
>
>    three locations for the port buffer is, that the DUT buffering scheme
> [O] is, that
> [P] is because
> [R] grammar
>
>
>    can be unknown or untested, and therefore the indication of where the
>    buffer is located helps understand the buffer architecture and
> [O]  and therefore the indication of where the
>    buffer is located helps understand the buffer architecture
> [P] and so knowing the buffer location helps clarify the buffer
> architecture
> [R] readability
>
>    therefore the total buffer size. The Port Buffer Size is an
>    informational value that MAY be provided from the DUT vendor. It is
>    not a value that is tested by benchmarking. Benchmarking will be done
>    using the Maximum Port Buffer Size or Maximum Buffer Size
>    methodology.
>
>    Maximum Port Buffer Size: this is in most cases the same as the Port
> [O] this is in most cases the same
> [P] in most cases, this is the same
> [R] readability
>
>    Buffer Size. In certain switch architecture called SoC (switch on
>    chip), there is a concept of port buffer and shared buffer pool
> [O] there is a concept
> [R] ? in certain switch architecture there is a concept? Or is there an
> actual port buffer?
>
>    available for all ports. Maximum Port Buffer, defines the scenario of
>    a SoC buffer, where this amount in B (byte), KB (kilobyte), MB
>    (megabyte) or GB (gigabyte) would represent the sum of the port
>    buffer along with the maximum value of shared buffer this given port
>    can take. The Maximum Port Buffer Size needs to be expressed along
> [O] Maximum Port Buffer, defines the scenario of
>    a SoC buffer, where this amount in B (byte), KB (kilobyte), MB
>    (megabyte) or GB (gigabyte) would represent the sum of the port
>    buffer along with the maximum value of shared buffer this given port
>    can take.
> [P] Maximum Port Buffer, in terms of an SoC buffer, represents the sum of
> the port buffer and the maximum value of shared buffer allowed for this
> port, defined in terms of B (byte), KB (kilobyte), MB (megabyte), or GB
> (gigabyte).
> [R] readability - I think. I has a hard time parsing the previous sentence.
>
>    with the frame MTU used for the measurement and the cos or dscp bit
>    value set for the test.
>
>    Example: a DUT has been measured to have 3KB of port buffer for 1518
>    frame size packets and a total of 4.7 MB of maximum port buffer for
>    1518 frame size packets and a cos of 0.
>
>    Maximum DUT Buffer Size: this is the total size of Buffer a DUT can
>    be measured to have. It is most likely different than the Maximum
>
> [O] It is most likely different than
> [P] either: It is, most likely, different to [OR] It is probably different
> to
> [R] grammar/readability
>
>    Port Buffer Size. It CAN also be different from the sum of Maximum
>    Port Buffer Size. The Maximum Buffer Size needs to be expressed along
>    with the frame MTU used for the measurement and along with the cos or
>    dscp value set during the test.
>
>    Example: a DUT has been measured to have 3KB of port buffer for 1518
>    frame size packets and a total of 4.7 MB of maximum port buffer for
>    1518 frame size packets. The DUT has a Maximum Buffer Size of 18 MB
>    at 1500 bytes and a cos of 0.
>
>    Burst: The burst is a fixed number of packets sent over a percentage
>    of linerate of a defined port speed. The amount of frames sent are
>    evenly distributed across the interval T. A constant C, can be
> [O] interval T. A constant C,
> [P] interval, T. A constant, C,
> [R] clarity
>
>    defined to provide the average time between two consecutive packets
>    evenly spaced.
>
>
> 6.1.2 Discussion
>
>    When measuring buffering on a DUT, it is important to understand what
>    the behavior is for each port, and also for all ports as this will
>    provide an evidence of the total amount of buffering available on the
>    switch. The terms of buffer efficiency here helps one understand what
> [O] When measuring buffering on a DUT, it is important to understand what
>    the behavior is for each port, and also for all ports as this will
>    provide an evidence of the total amount of buffering available on the
>    switch.
> [P] When measuring buffering on a DUT, it is important to understand the
> behavior for each and all ports. This provides data for the total amount of
> buffering available on the switch.
>
>    is the optimum packet size for the buffer to be used, or what is the
>    real volume of buffer available for a specific packet size. This
> [O] what is the optimum packet size for the buffer to be used, or what is
> the
>    real volume of buffer available for a specific packet size.
> [P] the optimum packet size for the buffer, or the real volume of the
> buffer available for a specific packet size
> [R] readability
>
>    section does not discuss how to conduct the test methodology, it
>    rather explains the buffer definitions and what metrics should be
> [O] , it rather explains
> [P] ; instead, it explains
>    provided for a comprehensive data center device buffering
>    benchmarking.
>
> 6.1.3 Measurement Units
>
>    When Buffer is measured:-the buffer size MUST be measured-the port
>    buffer size MAY be provided for each port-the maximum port buffer
>    size MUST be measured-the maximum DUT buffer size MUST be measured-
>    the intensity of microburst MAY be mentioned when a microburst test
>    is performed-the cos or dscp value set during the test SHOULD be
>    provided
>
> [O] entire paragraph above
> [R] this is very difficult to read with the hyphens. Guessing this was
> intended to be a list, but the XML got munged?
>
> 6.2 Incast
> 6.2.1 Definition
>
>    The term Incast, very commonly utilized in the data center, refers to
>    the traffic pattern of many-to-one or many-to-many conversations.
>    Typically in the data center it would refer to many different ingress
>    server ports(many), sending traffic to a common uplink (one), or
> [O] ports(many)
> [P] ports (many)
>
>    multiple uplinks (many). This pattern is generalized for any network
>    as many incoming ports sending traffic to one or few uplinks. It can
>    also be found in many-to-many traffic patterns.
>
>
> 6.2.2 Discussion
>
>
>    In this scenario, buffers are solicited on the DUT. In a ingress
>    buffering mechanism, the ingress port buffers would be solicited
>    along with Virtual Output Queues, when available; whereas in an
>    egress buffer mechanism, the egress buffer of the one outgoing port
>    would be used.
>
>    In either cases, regardless of where the buffer memory is located on
>    the switch architecture; the Incast creates buffer utilization.
> [O] In either cases, regardless of where the buffer memory is located on
>    the switch architecture; the Incast creates buffer utilization.
> [P] In either case, regardless of where the buffer memory is located in
> the switch architecture, the Incast creates the buffer utilization.
> [R] grammar
>
>
>    When one or more frames having synchronous arrival times at the DUT
>    they are considered forming an incast.
> [O] incast
> [P] Incast
> [R] consistency
>
>
> 7 Application Throughput: Data Center Goodput
>
> 7.3. Measurement Units
>
>    Example: a TCP file transfer over HTTP protocol on a 10Gb/s media.
>    The file cannot be transferred over Ethernet as a single continuous
>    stream. It must be broken down into individual frames of 1500 bytes
>    when the standard MTU [Maximum Transmission Unit] is used. Each
>    packet requires 20 bytes of IP header information and 20 bytes of TCP
>    header information, therefore 1460 byte are available per packet for
> [O] information, therefore
> [P] information; therefore,
> [R] grammar
>
>    the file transfer. Linux based systems are further limited to 1448
>    bytes as they also carry a 12 byte timestamp. Finally, the date is
>    transmitted in this example over Ethernet which adds a 26 byte
>    overhead per packet.
>
>
>
> --
> I don't think the execution is relevant when it was obviously a bad idea
> in the first place.
> This is like putting rabid weasels in your pants, and later expressing
> regret at having chosen those particular rabid weasels and that pair of
> pants.
>    ---maf
>