Re: [bmwg] Comments on draft-constantine-bmwg-traffic-management-01

["MORTON JR., ALFRED C (AL)" <[e-mail]>]

Thanks for your quick reply, Barry.

After reviewing section 5.2.1, I think you might benefit from using the term
TCP connections when you say TCP pattern in a few cases, but in
any event we need to define the terms and maybe add a few to describe
the test traffic, which includes transport and  application layers.

There are a couple of drafts you and your co-authors should look at:
draft-ietf-bmwg-ca-bench-meth-04 for some traffic generation/description ideas,
and
http://tools.ietf.org/html/draft-ietf-ippm-model-based-metrics-00
where there's some overlap -- the good kind where we can learn from each other.

regards,
Al (as a participant)

________________________________________
From: Barry Constantine [[e-mail]]
Sent: Sunday, July 28, 2013 7:40 AM
To: MORTON JR., ALFRED C (AL); [e-mail]
Subject: RE: Comments on draft-constantine-bmwg-traffic-management-01

Hi Al,

Expressing Buffer Delay in msec and percent makes good sense and we will add that to the next revision.

We also added a bullet  in the Berlin slide presentation to review the concept of TCP / Application test patterns (name bounces between TCP and Application).

Thank you,
Barry Constantine

JDSU Communications Test
Principal Member Technical Staff
301-325-7069

-----Original Message-----
From: MORTON JR., ALFRED C (AL) [mailto:[e-mail]]
Sent: Sunday, July 28, 2013 7:37 AM
To: Barry Constantine; [e-mail]
Subject: RE: Comments on draft-constantine-bmwg-traffic-management-01

Hi Barry and co-authors,

I'll need to take a look at 5.2.1 for the details on TCP patterns, to see if it resolves my concern.  Regarding the Buffer delay comment:

Al wrote:
...
Also, Buffer Delay is a bit odd expressed as a percentage of the minimum/baseline (to me at least). Maybe use a different name for this.
**************

you wrote:
...
We do not understand why you think Buffer Delay is odd; the percentage clearly shows the increase in delay under load due to queuing / shaping and many folks have found this useful in RFC 6349 test scenarios.  Perhaps we misunderstand your point?

Al replies:
It just seems to me that the real values of delay in miliseconds are more useful than the combined values expressed as a percentage, as you would need at least one of the real values to interpret the precentage and do some e2e path engineering in milliseconds.
Both the real values in milliseconds and the percentage could be reported, if folks cans stand a little redundancy, but IMO the technical audience needs the metric in expressed milliseconds.


________________________________________
From: Barry Constantine [[e-mail]]
Sent: Wednesday, July 24, 2013 11:11 AM
To: MORTON JR., ALFRED C (AL); [e-mail]
Subject: RE: Comments on draft-constantine-bmwg-traffic-management-01

Hi Al,

Thank you for all of the excellent comments and we will incorporate those as well as others received after the Berlin meeting.

Couple of points I wanted to clarify here though:

**************
   - TCP Test Pattern Execution Time (TTPET): RFC 6349 defined the TCP
   Transfer Time for bulk transfers, which is simply the measured time
   to transfer bytes across single or concurrent TCP connections. The
   TCP test patterns used in traffic management tests will be bulk
   transfer and interactive in nature; these test patterns simulate
   delay-tolerant applications like FTP, streaming video etc. The TTPET
   will be the measure of the time for a single execution of a TCP Test
   Pattern (TTP).  Average, minimum, and maximum times will be measured.

Comment: the use of the term "Test Pattern" in connection with TCP strikes me as odd.
A test pattern usually specifies some aspects of the packet stream that will be repeated. Instead, this is a bulk transfer where I assume the eventual traffic pattern will be determined dynamically by TCP's flow control.
***************

The paragraph probably caused confusion, it specifies that:
.  TCP test patterns used in traffic management tests will be bulk
   transfer and interactive in nature; these test patterns simulate
   delay-tolerant applications like FTP, streaming video etc

This includes interactive such as HTTP business applications, database applications, etc which are not bulk transfer and can be modeled in terms of the stateful packet stream (back and forth traffic).  "5.2.1. TCP Test Pattern Definitions" is a pretty lengthy discussion concerning the nature of TCP application traffic and some modeling references.  Let us know if clarifying the paragraph you highlighted with some text and references back to 5.2.1 would satisfy your comment.

**************
Comment/Question:  Without checking RFC6349 (because I'm out of time), are Retransmitted Packets included in the RTT calculations? It seems that you want to avoid them in order to infer the physical buffer time.

Also, Buffer Delay is a bit odd expressed as a percentage of the minimum/baseline (to me at least). Maybe use a different name for this.
**************

RFC 6349 does not explicitly state that retransmitted packets should not be included in RTT calculation, but by the nature of TCP, RTT does not make any sense except for the packet that is successfully ACKed.  But I do agree with your comment, it should be clarified in this new work; we'll reference RFC 6349 with the clarification.

We do not understand why you think Buffer Delay is odd; the percentage clearly shows the increase in delay under load due to queuing / shaping and many folks have found this useful in RFC 6349 test scenarios.  Perhaps we misunderstand your point?

Thanks again,
Barry



Thank you,
Barry Constantine

JDSU Communications Test
Principal Member Technical Staff
301-325-7069


-----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf Of MORTON JR., ALFRED C (AL)
Sent: Tuesday, July 09, 2013 11:47 AM
To: [e-mail]
Subject: [bmwg] Comments on draft-constantine-bmwg-traffic-management-01

Barry, Tim, and Ram,

I have some comments to share on the first part of your draft, below. There's no need to react or edit immediately, we can discuss further in Berlin...

Al
(as a participant)

-=-=-=-=-=-=-=-=-=-

Section 1.1 Traffic Management Overview

This section contains a brief description of key functions of a traffic management system. It would help to link these descriptions with references that provide more details, since I'm sure you have some at hand.

Also, BMWG has already defined some relevant terms in
http://tools.ietf.org/html/rfc4689
and it would be good to re-use that work where possible.

A specific suggestion:
OLD
  - Active Queue Management (AQM): monitors the status of internal
    queues and actively drops packets, which causes the sending hosts
        to back-off and in turn can alleviate queue congestion.
NEW
  - Active Queue Management (AQM): monitors the status of internal
    queues and actively drops (or re-marks) packets, which causes hosts using congestion-aware protocols
                              ^^^^^^^^^^^^^                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        to back-off and in turn can alleviate queue congestion.

and mention active queue management explicitly in Figure 1, like the other functions.
Re-marking or ECN and similar functions might be described separately.


Section 1.2

...
   For an example: a device is specified to be capable of shaping on all
   of it's egress ports.  The individual verification test would first be
   conducted to benchmark the advertised shaping function to specification.
   Then the capacity test would be executed to test the shaping function
   concurrently on all interfaces and with maximum traffic load.

s/it's/its/

and
   ...
   concurrently on all interfaces and with N instances of the shaping function and maximum traffic load.
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(is this a test that's planned?)


...
   Also note that the Network Delay Emulator (NDE) should be passive in
   nature such as a fiber spool.  This is recommended to eliminate the
   ...
   performance benchmarked independently.  This requirement will vary
   from test to test on desired traffic speed and should be calibrated
   before any test requiring delay, which can add a significant
   additional amount of testing to each step.

Question: is there no high-end capacity test for the NDE that would cover all traffic levels below? (I suppose that the degree of packet inspection performed by the emulator influences the result, but there must be some simplification possible here - fiber spools are not easily configured, etc.)


Section 3
...
   Within this framework, the metrics are defined for each traffic
   management test but do not include pass / fail criterion, which is
   not within the charter of BMWG.  This framework provides the test
   methods and metrics to conduct repeatable testing, which will
   provide the means to compare advertised performance between vendors.

Suggest:
   ...
   provide the means to compare measured performance between DUTs.
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...
   As mentioned in section 1.2, this framework describes the individual
   tests and metrics for several management functions.  It is also within
   scope that this framework will benchmark each function in terms of overall
   rated capacity.  This involves concurrent testing of multiple
   interfaces with the specific traffic management function enabled, and
   doing so to the capacity limit of each interface.

Suggest:
   ...
   interfaces with the specific traffic management function enabled,
   up to the capacity limit of each interface.
   ^^

...

   However, it is not within scope of this framework to test multiple
   traffic management functions concurrently (e.g. police on ingress and
   shape on egress ports).  The multitudes of possible combinations is
   almost unbounded and the ability to identify functional "break points"
   would be most times impossible.

Comment: While avoiding all possible combinations, it seems that usual profiles of concurrent functions would be useful to benchmark, IF the tests produce reliable and repeatable results. The wg participants could help by suggesting feature combinations.

...
   A goal of this framework is to define specific stateless traffic
   ("packet blasting") tests to conduct the benchmark tests and also to
   derive stateful test patterns (TCP or application layer) that can
   also be used to further benchmark the performance of applicable
   traffic management techniques such as shaping and congestion
   management techniques such as RED/WRED.  In cases where the network
   device is stateful in nature (i.e. firewall, etc.), stateful test
   pattern traffic is important to test along with UDP traffic in
   specific test scenarios (i.e. TCP application and UDP VoIP, etc.)

Suggest:
   ...
   specific test scenarios (i.e. applications using TCP transport and UDP VoIP, etc.)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Section 4.1 (for stateless/UDP)
...
   Stateless traffic measurements require that sequence number and
   time-stamp be inserted into the payload for lost packet and delay
   analyses.  This framework does not specify the packet format to carry
   sequence number of timing information.

Comment: This is very implementation-specific. The requirement is to provide some means to establish correspondence between sent and received packets, which could be a sequence number alone with time stamps stored elsewhere.

...
   - Burst Size Achieved (BSA): for the traffic policing and network
   queue tests, the tester will be configured to send bursts to test
   either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of
   a policer or the queue / buffer size configured in the DUT.  The
   Burst Size Achieved metric is a measure of the actual burst size
   received at the egress port of the DUT with no lost packets.  As an
   example, the configured CBS of a DUT is 64KB and after the burst test,
   only a 63 KB can be achieved without packet loss.  Then 63KB is the
   BSA.
Comments:
  - no requirement on Latency?
  - re-marking may replace loss

...
   - Lost Packets (LP): For all traffic management tests, the tester will
   transmit the test packets into the DUT ingress port and the number of
   packets received at the egress port will be measured.  The difference
   between packets transmitted into the ingress port and received at the
   egress port is the number of lost packets as measured at the egress
   port.  These packets must have unique identifiers such that only the
   test packets are measured.  RFC 4737 and RFC 2680 specify techniques
   to establish the threshold when a packet is lost versus out-of-order.
Suggest: Last sentence is too convoluted...

   ... RFC 2680 describes the need
   to establish the threshold to designate a packet as lost, and the
   threshold used MUST be reported with the results.

...
   - Out of Sequence (OOS): in additions to LF metric, the test
   packets must be monitored for sequence and the out-of-sequence (OOS)
   packets will be counted per RFC 4737 and RFC 2680.
Comment: Sequence is different from Order, and only Order evaluation is orthogonal to Loss. See RFC 4737 for a simple definition of "Reordered Packet".

...
   - Packet Delay Variation (PDV): the Packet Delay Variation metric is
   the variation between the timestamp of the received egress port
   packets and specified in RFC 1889.

Comment: since you have One-way Delay (PD defined above PDV), why resort to the RFC 1889 definition?  What does this give you?
Suggest: Specify PDV as defined in http://tools.ietf.org/html/rfc5481#section-4.2 instead, for the many reasons given in http://tools.ietf.org/html/rfc5481#page-27
(or call this IPDV to avoid confusion).

Section 4.2 (for TCP)
...
   - TCP Test Pattern Execution Time (TTPET): RFC 6349 defined the TCP
   Transfer Time for bulk transfers, which is simply the measured time
   to transfer bytes across single or concurrent TCP connections. The
   TCP test patterns used in traffic management tests will be bulk
   transfer and interactive in nature; these test patterns simulate
   delay-tolerant applications like FTP, streaming video etc. The TTPET
   will be the measure of the time for a single execution of a TCP Test
   Pattern (TTP).  Average, minimum, and maximum times will be measured.

Comment: the use of the term "Test Pattern" in connection with TCP strikes me as odd.
A test pattern usually specifies some aspects of the packet stream that will be repeated. Instead, this is a bulk transfer where I assume the eventual traffic pattern will be determined dynamically by TCP's flow control.
Isn't this simply "TCP Bulk Transfer Time" ?

...
   - TCP Efficiency: after the execution of the TCP Test Pattern, TCP
   Efficiency represents the percentage of Bytes that were not
   retransmitted.

                          Transmitted Bytes - Retransmitted Bytes

      TCP Efficiency % =  ---------------------------------------  X 100

                                   Transmitted Bytes

   Transmitted Bytes are the total number of TCP Bytes to be transmitted
   including the original and the retransmitted Bytes.
Comment: Here, I think you need to carefully define "Retransmitted Bytes"
or the door will be open to some "specsmanship". For example, if bytes are only received once at the egress, were they retransmitted?  Perhaps they are just bytes from Reordered packets...

...
     - Buffer Delay: represents the increase in RTT during a TCP test
     versus the baseline DUT RTT (non congested, inherent latency).  RTT
         and the technique to measure RTT (average versus baseline) are defined
         in RFC 6349.  Referencing RFC 6349, the average RTT is derived from
         the total of all measured RTTs during the actual test at every second
         divided by the test duration in seconds.

                                         Total RTTs during transfer
         Average RTT during transfer = -----------------------------
                                        Transfer duration in seconds


                        Average RTT during Transfer - Baseline RTT
       Buffer Delay % = ------------------------------------------ X 100
                                    Baseline RTT

Comment/Question:  Without checking RFC6349 (because I'm out of time), are Retransmitted Packets included in the RTT calculations? It seems that you want to avoid them in order to infer the physical buffer time.

Also, Buffer Delay is a bit odd expressed as a percentage of the minimum/baseline (to me at least). Maybe use a different name for this.

_______________________________________________
bmwg mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/bmwg