[bmwg] Comments on draft-ietf-bmwg-ca-bench-meth-01.txt

[Barry Constantine <[e-mail]>]

Hi Mike / Sarah,

The document is really coming along nicely.

Here are some comments / suggestions I had:

4.1.4.3.  Packet Loss

   The test tool SHOULD report the number of flow packets lost or
   dropped from source to destination.

>>>>>>>>>>>>
   The TCP Efficiency metric from RFC 6349 could be beneficial in this area.
   Specifically TCP Efficiency is the ratio of:

   total transmitted bytes - retransmitted bytes
   ------------------------------------------------------  x 100
                total transmitted bytes

  Ideal score is 100%.

   With TCP based application flows, it is the number of retransmitted bytes
   that is more telling than the number of lost packets, and the efficiency
   metric helps to quantify the overall performance.

   So I would consider measuring packet loss and TCP Efficiency (and on a per
   flow basis would be very useful as well).


 4.1.4.4.  Application Flow Latency

   The test tool SHOULD report the minimum, maximum and average amount
   of time an application flow member takes to traverse the DUT, as
   defined by RFC 1242 [3], Section 3.8.  This rate SHOULD be reported
   individually for each application protocol present within the traffic
   mix.

>>>>>>>>>>>>>>
   Similarly, the Buffer Delay Percent from RFC 6349 could be useful here.
   Buffer Delay =

   RTT (under load) - RTT baseline
   --------------------------------------- x 100
            RTT baseline

Ideal = 0%

Versus just min, max, and average, Buffer Delay percent (in combination with
TCP Efficiency) can segment performance issues into the loss related issues versus
device buffering performance issues.

I was involved with a benchmark of WAN Acceleration devices recently, and the
vendor found these metrics to be very beneficial for the reasons I stated above.

Thank you,
Barry Constantine