[bmwg] AD review of draft-ietf-bmwg-dcbench-terminology.

[Warren Kumari <[e-mail]>]

Hi there,

Apologies, I had forgotten to click the AD eval button... {DONE}

Thanks for a (generally) clear and useful document.

I have completed my AD review of draft-ietf-bmwg-dcbench-terminology and I
have a number of suggested edits to make the document even more readable /
clearer. In a few instances I wasn't quite sure what the document was
trying to say, so please let me know if any of the suggestions don't make
sense, or change the text meaning...

Also, please poke me once updated, so that I can progress it to IETF LC.

W
------

Abstract

The purpose of this informational document is to establish definitions,
discussion and measurement techniques for data center benchmarking.
[O] establish definitions, discussion and measurement techniques
[R] I'm not sure what is meant by "discussion" here. "Discussion and
measurement techniques" doesn't make much sense to me. Perhaps "to
establish definitions, foster discussion and describe measurement
techniques ..."?



Also, it is to introduce new terminologies applicable to data center
[O] benchmarking. Also, it is to introduce
[P] benchmarking, as well as introduce
[R] clarity/no need for a separate sentence.

performance evaluations. The purpose of this document is not to define
the test methodology, but rather establish the important concepts when
one is interested in benchmarking network switches and routers in the
[O] when one is interested in benchmarking
[P] for benchmarking
[R] readability

data center.



1.  Introduction

   Traffic patterns in the data center are not uniform and are contently
[O] contently
[P] constantly
[R] word choice/typo

   changing. They are dictated by the nature and variety of applications
   utilized in the data center. It can be largely east-west traffic
   flows in one data center and north-south in another, while some may
   combine both. Traffic patterns can be bursty in nature and contain
   many-to-one, many-to-many, or one-to-many flows. Each flow may also
   be small and latency sensitive or large and throughput sensitive
   while containing a mix of UDP and TCP traffic. All of which can
   coexist in a single cluster and flow through a single network device
   all at the same time. Benchmarking of network devices have long used
[O] All of which can
   coexist in a single cluster and flow through a single network device
   all at the same time.
[P] One or more of these may coexist in a single cluster and flow through a
single network device simultaneously.
[R] grammar/readability - unless this changes the intent?

   [RFC1242], [RFC2432], [RFC2544], [2] and [3]. These benchmarks have
   largely been focused around various latency attributes and max
   throughput of the Device Under Test being benchmarked. These
   standards are good at measuring theoretical max throughput,
   forwarding rates and latency under testing conditions, but to not
[O] , but to not
[P]; but they do not
[R] grammar

   represent real traffic patterns that may affect these networking
   devices. The data center networking devices covered are switches and
   routers.



1.2. Definition format

   Term to be defined. (e.g., Latency)

   Definition: The specific definition for the term.

   Discussion: A brief discussion about the term, it's application and
[O] it's
[P] its
[R] Grammar. Should be a possessive, not a contraction of "it is."
   any restrictions on measurement procedures.

   Measurement Units: Methodology for the measure and units used to
   report measurements of this term, if applicable.


2.  Latency


2.1. Definition

   Latency is a the amount of time it takes a frame to transit the DUT.
   Latency is measured in unit of time (seconds, milliseconds,
   microseconds and so on). The purpose of measuring latency is to
   understand what is the impact of adding a device in the communication
[O]  understand what is the impact
[P] understand the impact
[R] grammar

   path.

   The Latency interval can be assessed between different combinations
   of events, irrespectively of the type of switching device (bit
[O] irrespectively
[P] regardless
[R] grammar

   forwarding aka cut-through or store forward type of device)



2.2 Discussion

   FILO is the most important measuring definition. Any type of switches
[O] Any type of switches
[P] All switches
[R] readability

   MUST be measured with the FILO mechanism: FILO will include the
   latency of the switch and the latency of the frame as well as the
   serialization delay. It is a picture of the 'whole' latency going
   through the DUT. For applications, which are latency sensitive and
   can function with initial bytes of the frame, FIFO MAY be an
   additional type of measuring to supplement FILO.


3 Jitter
3.2 Discussion

   In addition to PDV Range and or a high percentile of PDV, Inter-
[O]  and or
[P] and/or

   Packet Delay Variation (IPDV) as defined in section 4.1 of RFC5481
   (differences between two consecutive packets) MAY be used for the
   purpose of determining how packet spacing has changed during
   transfer, for example to see if packet stream has become closely-
[O] for example to see
[P] for example, to see
[R] grammar

   spaced or "bursty". However, the Absolute Value of IPDV SHOULD NOT be
   used, as this collapses the "bursty" and "dispersed" sides of the
   IPDV distribution together.

4 Physical Layer Calibration
4.2 Discussion

   Physical layer calibration is part of the end to end latency, which
   should be taken into acknowledgment while evaluating the DUT. Small
   variations of the physical components of the test may impact the
   latency being measure so they MUST be described when presenting
[O] latency being measure so
[P] latency being measured, so
[R] grammar

   results.


5 Line rate

5.1 Definition

   The transmit timing, or maximum transmitted data rate is controlled
   by the "transmit clock" in the DUT.  The receive timing (maximum
   ingress data rate) is derived from the transmit clock of the
   connected interface.

   The line rate or physical layer frame rate is the maximum capacity to
   send frames of a specific size at the transmit clock frequency of the
   DUT.

   The term port capacity term defines the maximum speed capability for
[O] The term port capacity term
[P] The term "port capacity"
[R] readability

   the given port; for example 1GE, 10GE, 40GE, 100GE etc.

   The frequency ("clock rate") of the transmit clock in any two
   connected interfaces will never be precisely the same, therefore a
   tolerance is needed, this will be expressed by Parts Per Million
[O] same, therefore a tolerance is needed, this will
[P] same; therefore, a tolerance is needed. This will
[R] grammar


5.2 Discussion

   For a transmit clock source, most Ethernet switches use "clock
   modules" (also called "oscillator modules") that are sealed,
   internally temperature-compensated, and very accurate. The output
   frequency of these modules is not adjustable because it is not
   necessary.  Many test sets, however, offer a software-controlled
   adjustment of the transmit clock rate, which should be used to
   compensate the test equipment to not send more than line rate of the
   DUT.

[O] which should be used to
   compensate the test equipment to not send more than line rate of the
   DUT.
[R] please reword for clarity.

   To allow for the minor variations typically found in the clock rate
   of commercially-available clock modules and other crystal-based


   Test set equipment manufacturers are well-aware of the standards, and
   allows a software-controlled +/- 100 PPM "offset" (clock-rate
[O] allows
[P] allow
[R] grammar

   adjustment) to compensate for normal variations in the clock speed of
   "devices under test". This offset adjustment allows engineers to
   determine the approximate speed the connected device is operating,
   and verify that it is within parameters allowed by standards.



5.3 Measurement Units


   In a production network, it is very unlikely to see precise line rate
   over a very brief period. There is no observable difference between
   dropping packets at 99% of line rate and 100% of line rate. -Line
   rate CAN measured at 100% of line rate with a -100PPM adjustment. -
   Line rate SHOULD be measured at 99,98% with 0 PPM adjustment.-The PPM
[O] .-The
[P] . The
[R] Typo Or was this intended to be a list?

   adjustment SHOULD only be used for a line rate type of measurement


6  Buffering
6.1 Buffer
6.1.1 Definition

   Buffer Size: the term buffer size, represents the total amount of
[O] the term buffer size, represents
[P] the term buffer size represents
[R] grammar

   frame buffering memory available on a DUT. This size is expressed in
   Byte; KB (kilobytes), MB (megabytes) or GB (gigabyte). When the
   buffer size is expressed it SHOULD be defined by a size metric
   defined above. When the buffer size is expressed, an indication of
[O]  defined above
[P] listed above [OR] stated above
[R] the sizes are not "defined" above

   the frame MTU used for that measurement is also necessary as well as
   the cos or dscp value set; as often times the buffers are carved by
   quality of service implementation. (please refer to the buffer
[O] . (please
[P] . (Please
[R] grammar

   efficiency section for further details).

   Example: Buffer Size of DUT when sending 1518 bytes frames is 18 Mb.

   Port Buffer Size: the port buffer size is the amount of buffer a
   single ingress port, egress port or combination of ingress and egress
   buffering location for a single port. The reason of mentioning the
[O] reason of mentioning
[P] reason for mentioning
[R] grammar

   three locations for the port buffer is, that the DUT buffering scheme
[O] is, that
[P] is because
[R] grammar


   can be unknown or untested, and therefore the indication of where the
   buffer is located helps understand the buffer architecture and
[O]  and therefore the indication of where the
   buffer is located helps understand the buffer architecture
[P] and so knowing the buffer location helps clarify the buffer architecture
[R] readability

   therefore the total buffer size. The Port Buffer Size is an
   informational value that MAY be provided from the DUT vendor. It is
   not a value that is tested by benchmarking. Benchmarking will be done
   using the Maximum Port Buffer Size or Maximum Buffer Size
   methodology.

   Maximum Port Buffer Size: this is in most cases the same as the Port
[O] this is in most cases the same
[P] in most cases, this is the same
[R] readability

   Buffer Size. In certain switch architecture called SoC (switch on
   chip), there is a concept of port buffer and shared buffer pool
[O] there is a concept
[R] ? in certain switch architecture there is a concept? Or is there an
actual port buffer?

   available for all ports. Maximum Port Buffer, defines the scenario of
   a SoC buffer, where this amount in B (byte), KB (kilobyte), MB
   (megabyte) or GB (gigabyte) would represent the sum of the port
   buffer along with the maximum value of shared buffer this given port
   can take. The Maximum Port Buffer Size needs to be expressed along
[O] Maximum Port Buffer, defines the scenario of
   a SoC buffer, where this amount in B (byte), KB (kilobyte), MB
   (megabyte) or GB (gigabyte) would represent the sum of the port
   buffer along with the maximum value of shared buffer this given port
   can take.
[P] Maximum Port Buffer, in terms of an SoC buffer, represents the sum of
the port buffer and the maximum value of shared buffer allowed for this
port, defined in terms of B (byte), KB (kilobyte), MB (megabyte), or GB
(gigabyte).
[R] readability - I think. I has a hard time parsing the previous sentence.

   with the frame MTU used for the measurement and the cos or dscp bit
   value set for the test.

   Example: a DUT has been measured to have 3KB of port buffer for 1518
   frame size packets and a total of 4.7 MB of maximum port buffer for
   1518 frame size packets and a cos of 0.

   Maximum DUT Buffer Size: this is the total size of Buffer a DUT can
   be measured to have. It is most likely different than the Maximum

[O] It is most likely different than
[P] either: It is, most likely, different to [OR] It is probably different
to
[R] grammar/readability

   Port Buffer Size. It CAN also be different from the sum of Maximum
   Port Buffer Size. The Maximum Buffer Size needs to be expressed along
   with the frame MTU used for the measurement and along with the cos or
   dscp value set during the test.

   Example: a DUT has been measured to have 3KB of port buffer for 1518
   frame size packets and a total of 4.7 MB of maximum port buffer for
   1518 frame size packets. The DUT has a Maximum Buffer Size of 18 MB
   at 1500 bytes and a cos of 0.

   Burst: The burst is a fixed number of packets sent over a percentage
   of linerate of a defined port speed. The amount of frames sent are
   evenly distributed across the interval T. A constant C, can be
[O] interval T. A constant C,
[P] interval, T. A constant, C,
[R] clarity

   defined to provide the average time between two consecutive packets
   evenly spaced.


6.1.2 Discussion

   When measuring buffering on a DUT, it is important to understand what
   the behavior is for each port, and also for all ports as this will
   provide an evidence of the total amount of buffering available on the
   switch. The terms of buffer efficiency here helps one understand what
[O] When measuring buffering on a DUT, it is important to understand what
   the behavior is for each port, and also for all ports as this will
   provide an evidence of the total amount of buffering available on the
   switch.
[P] When measuring buffering on a DUT, it is important to understand the
behavior for each and all ports. This provides data for the total amount of
buffering available on the switch.

   is the optimum packet size for the buffer to be used, or what is the
   real volume of buffer available for a specific packet size. This
[O] what is the optimum packet size for the buffer to be used, or what is
the
   real volume of buffer available for a specific packet size.
[P] the optimum packet size for the buffer, or the real volume of the
buffer available for a specific packet size
[R] readability

   section does not discuss how to conduct the test methodology, it
   rather explains the buffer definitions and what metrics should be
[O] , it rather explains
[P] ; instead, it explains
   provided for a comprehensive data center device buffering
   benchmarking.

6.1.3 Measurement Units

   When Buffer is measured:-the buffer size MUST be measured-the port
   buffer size MAY be provided for each port-the maximum port buffer
   size MUST be measured-the maximum DUT buffer size MUST be measured-
   the intensity of microburst MAY be mentioned when a microburst test
   is performed-the cos or dscp value set during the test SHOULD be
   provided

[O] entire paragraph above
[R] this is very difficult to read with the hyphens. Guessing this was
intended to be a list, but the XML got munged?

6.2 Incast
6.2.1 Definition

   The term Incast, very commonly utilized in the data center, refers to
   the traffic pattern of many-to-one or many-to-many conversations.
   Typically in the data center it would refer to many different ingress
   server ports(many), sending traffic to a common uplink (one), or
[O] ports(many)
[P] ports (many)

   multiple uplinks (many). This pattern is generalized for any network
   as many incoming ports sending traffic to one or few uplinks. It can
   also be found in many-to-many traffic patterns.


6.2.2 Discussion


   In this scenario, buffers are solicited on the DUT. In a ingress
   buffering mechanism, the ingress port buffers would be solicited
   along with Virtual Output Queues, when available; whereas in an
   egress buffer mechanism, the egress buffer of the one outgoing port
   would be used.

   In either cases, regardless of where the buffer memory is located on
   the switch architecture; the Incast creates buffer utilization.
[O] In either cases, regardless of where the buffer memory is located on
   the switch architecture; the Incast creates buffer utilization.
[P] In either case, regardless of where the buffer memory is located in the
switch architecture, the Incast creates the buffer utilization.
[R] grammar


   When one or more frames having synchronous arrival times at the DUT
   they are considered forming an incast.
[O] incast
[P] Incast
[R] consistency


7 Application Throughput: Data Center Goodput

7.3. Measurement Units

   Example: a TCP file transfer over HTTP protocol on a 10Gb/s media.
   The file cannot be transferred over Ethernet as a single continuous
   stream. It must be broken down into individual frames of 1500 bytes
   when the standard MTU [Maximum Transmission Unit] is used. Each
   packet requires 20 bytes of IP header information and 20 bytes of TCP
   header information, therefore 1460 byte are available per packet for
[O] information, therefore
[P] information; therefore,
[R] grammar

   the file transfer. Linux based systems are further limited to 1448
   bytes as they also carry a 12 byte timestamp. Finally, the date is
   transmitted in this example over Ethernet which adds a 26 byte
   overhead per packet.



-- 
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
   ---maf