Re: [bmwg] Comments on draft-constantine-bmwg-traffic-management-01
Barry Constantine <Barry.Constantine@jdsu.com> Sun, 28 July 2013 12:58 UTC
Return-Path: <Barry.Constantine@jdsu.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 51DE521F9C2D for <bmwg@ietfa.amsl.com>; Sun, 28 Jul 2013 05:58:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.321
X-Spam-Level:
X-Spam-Status: No, score=-2.321 tagged_above=-999 required=5 tests=[AWL=-0.055, BAYES_00=-2.599, IP_NOT_FRIENDLY=0.334]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id deT59UbA+1sm for <bmwg@ietfa.amsl.com>; Sun, 28 Jul 2013 05:58:34 -0700 (PDT)
Received: from mx0b-00158d01.pphosted.com (mx0b-00158d01.pphosted.com [67.231.152.180]) by ietfa.amsl.com (Postfix) with ESMTP id 35D8321F9998 for <bmwg@ietf.org>; Sun, 28 Jul 2013 05:58:31 -0700 (PDT)
Received: from pps.filterd (m0043258.ppops.net [127.0.0.1]) by mx0b-00158d01.pphosted.com (8.14.5/8.14.5) with SMTP id r6SCwT1w003651; Sun, 28 Jul 2013 05:58:29 -0700
Received: from mx2.jdsu.com ([157.234.211.51]) by mx0b-00158d01.pphosted.com with ESMTP id 1dv8n0n9ps-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 28 Jul 2013 05:58:28 -0700
Received: from AMEXHTCA03.ds.jdsu.net (10.239.69.13) by mx2.jdsu.com (10.239.15.51) with Microsoft SMTP Server (TLS) id 14.2.342.3; Sun, 28 Jul 2013 05:58:44 -0700
Received: from AMEXMB01.ds.jdsu.net ([fe80::9402:2c4c:29f3:a264]) by AMEXHTCA03.ds.jdsu.net ([fe80::24df:4228:5274:253d%14]) with mapi id 14.02.0342.003; Sun, 28 Jul 2013 05:58:26 -0700
From: Barry Constantine <Barry.Constantine@jdsu.com>
To: "MORTON JR., ALFRED C (AL)" <acmorton@att.com>, "bmwg@ietf.org" <bmwg@ietf.org>
Thread-Topic: Comments on draft-constantine-bmwg-traffic-management-01
Thread-Index: Ac58rZlIZ9MQXqdTSMGJpshMwUaH+gLyaH/QAMNV6Z4AAJhb4AABuUQeAAD0gkA=
Date: Sun, 28 Jul 2013 12:58:26 +0000
Message-ID: <DE2AAE0A8826CF4ABC3A6CCB756356EB190DC2@AMEXMB01.ds.jdsu.net>
References: <F1312FAF1A1E624DA0972D1C9A91379A1CA22DE618@njfpsrvexg7.research.att.com>, <DE2AAE0A8826CF4ABC3A6CCB756356EB18D48F@AMEXMB01.ds.jdsu.net> <F1312FAF1A1E624DA0972D1C9A91379A1CA4C6DC2B@njfpsrvexg7.research.att.com>, <DE2AAE0A8826CF4ABC3A6CCB756356EB190CE3@AMEXMB01.ds.jdsu.net> <F1312FAF1A1E624DA0972D1C9A91379A1CA4C6DC2D@njfpsrvexg7.research.att.com>
In-Reply-To: <F1312FAF1A1E624DA0972D1C9A91379A1CA4C6DC2D@njfpsrvexg7.research.att.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [157.234.234.5]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8794, 1.0.431, 0.0.0000 definitions=2013-07-28_05:2013-07-26, 2013-07-28, 1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1305240000 definitions=main-1307280056
Subject: Re: [bmwg] Comments on draft-constantine-bmwg-traffic-management-01
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 28 Jul 2013 12:58:39 -0000
That sounds great Al, we meant to reference the ca-benchmarking work; I really liked the idea of representative application mixes listed in that document and that flowed into the thought process for this work. We'll take a look at the ippm work as well. Thank you, Barry Constantine JDSU Communications Test Principal Member Technical Staff 301-325-7069 -----Original Message----- From: MORTON JR., ALFRED C (AL) [mailto:acmorton@att.com] Sent: Sunday, July 28, 2013 8:35 AM To: Barry Constantine; bmwg@ietf.org Subject: RE: Comments on draft-constantine-bmwg-traffic-management-01 Thanks for your quick reply, Barry. After reviewing section 5.2.1, I think you might benefit from using the term TCP connections when you say TCP pattern in a few cases, but in any event we need to define the terms and maybe add a few to describe the test traffic, which includes transport and application layers. There are a couple of drafts you and your co-authors should look at: draft-ietf-bmwg-ca-bench-meth-04 for some traffic generation/description ideas, and http://tools.ietf.org/html/draft-ietf-ippm-model-based-metrics-00 where there's some overlap -- the good kind where we can learn from each other. regards, Al (as a participant) ________________________________________ From: Barry Constantine [Barry.Constantine@jdsu.com] Sent: Sunday, July 28, 2013 7:40 AM To: MORTON JR., ALFRED C (AL); bmwg@ietf.org Subject: RE: Comments on draft-constantine-bmwg-traffic-management-01 Hi Al, Expressing Buffer Delay in msec and percent makes good sense and we will add that to the next revision. We also added a bullet in the Berlin slide presentation to review the concept of TCP / Application test patterns (name bounces between TCP and Application). Thank you, Barry Constantine JDSU Communications Test Principal Member Technical Staff 301-325-7069 -----Original Message----- From: MORTON JR., ALFRED C (AL) [mailto:acmorton@att.com] Sent: Sunday, July 28, 2013 7:37 AM To: Barry Constantine; bmwg@ietf.org Subject: RE: Comments on draft-constantine-bmwg-traffic-management-01 Hi Barry and co-authors, I'll need to take a look at 5.2.1 for the details on TCP patterns, to see if it resolves my concern. Regarding the Buffer delay comment: Al wrote: ... Also, Buffer Delay is a bit odd expressed as a percentage of the minimum/baseline (to me at least). Maybe use a different name for this. ************** you wrote: ... We do not understand why you think Buffer Delay is odd; the percentage clearly shows the increase in delay under load due to queuing / shaping and many folks have found this useful in RFC 6349 test scenarios. Perhaps we misunderstand your point? Al replies: It just seems to me that the real values of delay in miliseconds are more useful than the combined values expressed as a percentage, as you would need at least one of the real values to interpret the precentage and do some e2e path engineering in milliseconds. Both the real values in milliseconds and the percentage could be reported, if folks cans stand a little redundancy, but IMO the technical audience needs the metric in expressed milliseconds. ________________________________________ From: Barry Constantine [Barry.Constantine@jdsu.com] Sent: Wednesday, July 24, 2013 11:11 AM To: MORTON JR., ALFRED C (AL); bmwg@ietf.org Subject: RE: Comments on draft-constantine-bmwg-traffic-management-01 Hi Al, Thank you for all of the excellent comments and we will incorporate those as well as others received after the Berlin meeting. Couple of points I wanted to clarify here though: ************** - TCP Test Pattern Execution Time (TTPET): RFC 6349 defined the TCP Transfer Time for bulk transfers, which is simply the measured time to transfer bytes across single or concurrent TCP connections. The TCP test patterns used in traffic management tests will be bulk transfer and interactive in nature; these test patterns simulate delay-tolerant applications like FTP, streaming video etc. The TTPET will be the measure of the time for a single execution of a TCP Test Pattern (TTP). Average, minimum, and maximum times will be measured. Comment: the use of the term "Test Pattern" in connection with TCP strikes me as odd. A test pattern usually specifies some aspects of the packet stream that will be repeated. Instead, this is a bulk transfer where I assume the eventual traffic pattern will be determined dynamically by TCP's flow control. *************** The paragraph probably caused confusion, it specifies that: . TCP test patterns used in traffic management tests will be bulk transfer and interactive in nature; these test patterns simulate delay-tolerant applications like FTP, streaming video etc This includes interactive such as HTTP business applications, database applications, etc which are not bulk transfer and can be modeled in terms of the stateful packet stream (back and forth traffic). "5.2.1. TCP Test Pattern Definitions" is a pretty lengthy discussion concerning the nature of TCP application traffic and some modeling references. Let us know if clarifying the paragraph you highlighted with some text and references back to 5.2.1 would satisfy your comment. ************** Comment/Question: Without checking RFC6349 (because I'm out of time), are Retransmitted Packets included in the RTT calculations? It seems that you want to avoid them in order to infer the physical buffer time. Also, Buffer Delay is a bit odd expressed as a percentage of the minimum/baseline (to me at least). Maybe use a different name for this. ************** RFC 6349 does not explicitly state that retransmitted packets should not be included in RTT calculation, but by the nature of TCP, RTT does not make any sense except for the packet that is successfully ACKed. But I do agree with your comment, it should be clarified in this new work; we'll reference RFC 6349 with the clarification. We do not understand why you think Buffer Delay is odd; the percentage clearly shows the increase in delay under load due to queuing / shaping and many folks have found this useful in RFC 6349 test scenarios. Perhaps we misunderstand your point? Thanks again, Barry Thank you, Barry Constantine JDSU Communications Test Principal Member Technical Staff 301-325-7069 -----Original Message----- From: bmwg-bounces@ietf.org [mailto:bmwg-bounces@ietf.org] On Behalf Of MORTON JR., ALFRED C (AL) Sent: Tuesday, July 09, 2013 11:47 AM To: bmwg@ietf.org Subject: [bmwg] Comments on draft-constantine-bmwg-traffic-management-01 Barry, Tim, and Ram, I have some comments to share on the first part of your draft, below. There's no need to react or edit immediately, we can discuss further in Berlin... Al (as a participant) -=-=-=-=-=-=-=-=-=- Section 1.1 Traffic Management Overview This section contains a brief description of key functions of a traffic management system. It would help to link these descriptions with references that provide more details, since I'm sure you have some at hand. Also, BMWG has already defined some relevant terms in http://tools.ietf.org/html/rfc4689 and it would be good to re-use that work where possible. A specific suggestion: OLD - Active Queue Management (AQM): monitors the status of internal queues and actively drops packets, which causes the sending hosts to back-off and in turn can alleviate queue congestion. NEW - Active Queue Management (AQM): monitors the status of internal queues and actively drops (or re-marks) packets, which causes hosts using congestion-aware protocols ^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ to back-off and in turn can alleviate queue congestion. and mention active queue management explicitly in Figure 1, like the other functions. Re-marking or ECN and similar functions might be described separately. Section 1.2 ... For an example: a device is specified to be capable of shaping on all of it's egress ports. The individual verification test would first be conducted to benchmark the advertised shaping function to specification. Then the capacity test would be executed to test the shaping function concurrently on all interfaces and with maximum traffic load. s/it's/its/ and ... concurrently on all interfaces and with N instances of the shaping function and maximum traffic load. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (is this a test that's planned?) ... Also note that the Network Delay Emulator (NDE) should be passive in nature such as a fiber spool. This is recommended to eliminate the ... performance benchmarked independently. This requirement will vary from test to test on desired traffic speed and should be calibrated before any test requiring delay, which can add a significant additional amount of testing to each step. Question: is there no high-end capacity test for the NDE that would cover all traffic levels below? (I suppose that the degree of packet inspection performed by the emulator influences the result, but there must be some simplification possible here - fiber spools are not easily configured, etc.) Section 3 ... Within this framework, the metrics are defined for each traffic management test but do not include pass / fail criterion, which is not within the charter of BMWG. This framework provides the test methods and metrics to conduct repeatable testing, which will provide the means to compare advertised performance between vendors. Suggest: ... provide the means to compare measured performance between DUTs. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... As mentioned in section 1.2, this framework describes the individual tests and metrics for several management functions. It is also within scope that this framework will benchmark each function in terms of overall rated capacity. This involves concurrent testing of multiple interfaces with the specific traffic management function enabled, and doing so to the capacity limit of each interface. Suggest: ... interfaces with the specific traffic management function enabled, up to the capacity limit of each interface. ^^ ... However, it is not within scope of this framework to test multiple traffic management functions concurrently (e.g. police on ingress and shape on egress ports). The multitudes of possible combinations is almost unbounded and the ability to identify functional "break points" would be most times impossible. Comment: While avoiding all possible combinations, it seems that usual profiles of concurrent functions would be useful to benchmark, IF the tests produce reliable and repeatable results. The wg participants could help by suggesting feature combinations. ... A goal of this framework is to define specific stateless traffic ("packet blasting") tests to conduct the benchmark tests and also to derive stateful test patterns (TCP or application layer) that can also be used to further benchmark the performance of applicable traffic management techniques such as shaping and congestion management techniques such as RED/WRED. In cases where the network device is stateful in nature (i.e. firewall, etc.), stateful test pattern traffic is important to test along with UDP traffic in specific test scenarios (i.e. TCP application and UDP VoIP, etc.) Suggest: ... specific test scenarios (i.e. applications using TCP transport and UDP VoIP, etc.) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Section 4.1 (for stateless/UDP) ... Stateless traffic measurements require that sequence number and time-stamp be inserted into the payload for lost packet and delay analyses. This framework does not specify the packet format to carry sequence number of timing information. Comment: This is very implementation-specific. The requirement is to provide some means to establish correspondence between sent and received packets, which could be a sequence number alone with time stamps stored elsewhere. ... - Burst Size Achieved (BSA): for the traffic policing and network queue tests, the tester will be configured to send bursts to test either the Committed Burst Size (CBS) or Excess Burst Size (EBS) of a policer or the queue / buffer size configured in the DUT. The Burst Size Achieved metric is a measure of the actual burst size received at the egress port of the DUT with no lost packets. As an example, the configured CBS of a DUT is 64KB and after the burst test, only a 63 KB can be achieved without packet loss. Then 63KB is the BSA. Comments: - no requirement on Latency? - re-marking may replace loss ... - Lost Packets (LP): For all traffic management tests, the tester will transmit the test packets into the DUT ingress port and the number of packets received at the egress port will be measured. The difference between packets transmitted into the ingress port and received at the egress port is the number of lost packets as measured at the egress port. These packets must have unique identifiers such that only the test packets are measured. RFC 4737 and RFC 2680 specify techniques to establish the threshold when a packet is lost versus out-of-order. Suggest: Last sentence is too convoluted... ... RFC 2680 describes the need to establish the threshold to designate a packet as lost, and the threshold used MUST be reported with the results. ... - Out of Sequence (OOS): in additions to LF metric, the test packets must be monitored for sequence and the out-of-sequence (OOS) packets will be counted per RFC 4737 and RFC 2680. Comment: Sequence is different from Order, and only Order evaluation is orthogonal to Loss. See RFC 4737 for a simple definition of "Reordered Packet". ... - Packet Delay Variation (PDV): the Packet Delay Variation metric is the variation between the timestamp of the received egress port packets and specified in RFC 1889. Comment: since you have One-way Delay (PD defined above PDV), why resort to the RFC 1889 definition? What does this give you? Suggest: Specify PDV as defined in http://tools.ietf.org/html/rfc5481#section-4.2 instead, for the many reasons given in http://tools.ietf.org/html/rfc5481#page-27 (or call this IPDV to avoid confusion). Section 4.2 (for TCP) ... - TCP Test Pattern Execution Time (TTPET): RFC 6349 defined the TCP Transfer Time for bulk transfers, which is simply the measured time to transfer bytes across single or concurrent TCP connections. The TCP test patterns used in traffic management tests will be bulk transfer and interactive in nature; these test patterns simulate delay-tolerant applications like FTP, streaming video etc. The TTPET will be the measure of the time for a single execution of a TCP Test Pattern (TTP). Average, minimum, and maximum times will be measured. Comment: the use of the term "Test Pattern" in connection with TCP strikes me as odd. A test pattern usually specifies some aspects of the packet stream that will be repeated. Instead, this is a bulk transfer where I assume the eventual traffic pattern will be determined dynamically by TCP's flow control. Isn't this simply "TCP Bulk Transfer Time" ? ... - TCP Efficiency: after the execution of the TCP Test Pattern, TCP Efficiency represents the percentage of Bytes that were not retransmitted. Transmitted Bytes - Retransmitted Bytes TCP Efficiency % = --------------------------------------- X 100 Transmitted Bytes Transmitted Bytes are the total number of TCP Bytes to be transmitted including the original and the retransmitted Bytes. Comment: Here, I think you need to carefully define "Retransmitted Bytes" or the door will be open to some "specsmanship". For example, if bytes are only received once at the egress, were they retransmitted? Perhaps they are just bytes from Reordered packets... ... - Buffer Delay: represents the increase in RTT during a TCP test versus the baseline DUT RTT (non congested, inherent latency). RTT and the technique to measure RTT (average versus baseline) are defined in RFC 6349. Referencing RFC 6349, the average RTT is derived from the total of all measured RTTs during the actual test at every second divided by the test duration in seconds. Total RTTs during transfer Average RTT during transfer = ----------------------------- Transfer duration in seconds Average RTT during Transfer - Baseline RTT Buffer Delay % = ------------------------------------------ X 100 Baseline RTT Comment/Question: Without checking RFC6349 (because I'm out of time), are Retransmitted Packets included in the RTT calculations? It seems that you want to avoid them in order to infer the physical buffer time. Also, Buffer Delay is a bit odd expressed as a percentage of the minimum/baseline (to me at least). Maybe use a different name for this. _______________________________________________ bmwg mailing list bmwg@ietf.org https://www.ietf.org/mailman/listinfo/bmwg
- [bmwg] Comments on draft-constantine-bmwg-traffic… Gilles Forget
- [bmwg] Comments on draft-constantine-bmwg-traffic… MORTON JR., ALFRED C (AL)
- Re: [bmwg] Comments on draft-constantine-bmwg-tra… Barry Constantine
- Re: [bmwg] Comments on draft-constantine-bmwg-tra… MORTON JR., ALFRED C (AL)
- Re: [bmwg] Comments on draft-constantine-bmwg-tra… Barry Constantine
- Re: [bmwg] Comments on draft-constantine-bmwg-tra… MORTON JR., ALFRED C (AL)
- Re: [bmwg] Comments on draft-constantine-bmwg-tra… Barry Constantine