Re: [ippm] WGLC for draft-ietf-ippm-tcp-throughput-tm-06.txt

[Al Morton <[e-mail]>]

Henk, IPPM,

Henk wrote:
 > Please review the draft and raise any issues by Monday, September 20,
 > 9:00 UTC.  An URL for the draft is:
 >
 >     http://www.ietf.org/id/draft-ietf-ippm-tcp-throughput-tm-06.txt

I didn't remember the 9:00 UTC aspect, just that the 20th was the
deadline...

I have the following comments on  *-ippm-tcp-throughput-tm-06.txt
(written on the plane yesterday, so at least they were prepared on time).
There may be some overlap with earlier comments, I didn't have access
to all of them while reading.

Al

S1 - Intro

p1 current text:
>    Testing an operational network prior to customer activation is
>    referred to as "turn-up" testing and the SLA (Service Level
>    Agreement) is generally based upon Layer 2/3 packet throughput,
>    delay, loss and jitter.

I don't think this methodology can be the basis for SLA on throughput,
even TCP throughput, because of the many reasons for variability in
the results and the fact that IPPM is not writing standards track metrics
in this draft. I have further comments along these lines, but it would
suffice to replace throughput with information rate for now.
Other suggested edits below:

NEW:
    Testing an operational network prior to customer activation is
    referred to as "turn-up" testing and the SLA (Service Level
    Agreement) is generally based upon Layer 2/3 information rate,
    packet delay, loss and delay variation.      ^^^^^^^^^^^^^^^^^
    ^^^^^^                 ^^^^^^^^^^^^^^^^

-------------------------------------------------------------------------
p7 current text:

>    One other important note, it is highly recommended that traditional
>    Layer 2/3 type tests are conducted to verify the integrity of the
>    network before conducting TCP tests.  Examples include RFC 2544
>    [RFC2544], iperf (UDP mode), or manual packet layer test techniques
>    where packet throughput, loss, and delay measurements are conducted.

There is definitely a need for this phase of testing, and I support
including this paragraph in the draft.

However, IPPM needs to be very careful with the wording here, because:

- RFC 2544 has been applied well-beyond its intended scope for
   operational service turn-up testing.  RFC 2544 is lab-use only.

It's a problem when there's a need, but no standard.
RFC 2544-like methods were applied to fill the gap.
And we've got that situation now with many manufacturers
pointing to RFC 2544 and doing something 2544-like, but not
really embracing the scope of 2544 procedures (they only work
reliably in the lab - the live network is different/wilder animal).
There is an effort intended to address the Service Activation
Test methodology in the Ethernet Services space, ITU-T Y.156sam,
a work in progress in Question 17 of Study Group 12.

Also, IETF working groups and the IESG will certainly respect the
scope of other RFCs and BMWG's scope in particular, because
if BMWG methods were let loose on networks, congestion and
other bad scenarios will result (especially without the built-in
safety net that BMWG provided - a testing-only address space that
routers can always drop if they see it).

So I suggest the following wording:
NEW:
    One other important note, it is highly recommended that traditional
    Layer 2/3 type tests are conducted to verify the integrity of the
    network before conducting TCP tests.  Examples include iperf (UDP mode)
    and manual packet layer test techniques
    where packet throughput, loss, and delay measurements are conducted.
    When available, standardized testing similar to RFC 2544 [RFC2544] but
    adapted for use on operational networks may be used (because RFC 2544
    methods are not intended for use outside the lab environment).

-----------------------------------------------------------------------

S2
OLD
2. Goals of this Methodology
suggest NEW
2. Scope and Goals of this Methodology

OLD
>    Before defining the goals of this methodology, it is important to
>    clearly define the areas that are not intended to be measured or
>    analyzed by such a methodology.
suggest NEW
    Before defining the goals of this methodology, it is important to
    clearly define the areas that are out-of-scope for this
    methodology.

suggest additional out-of-scope item:

The methodology is not intended to measure TCP throughput as part of a
SLA, or to compare the TCP performance between service providers or
to compare between implementations of this methodology (test equipment).

------------------------------------------------------------------------


2.2 Metrics for TCP Throughput Tests
...
>    The TCP Transfer time can also be used to provide a normalized ratio
>    of the actual TCP Transfer Time versus ideal Transfer Time.  This
>    ratio is called the TCP Transfer Index and is defined as:
>
>                      Actual TCP Transfer Time
>                     -------------------------
>                      Ideal TCP Transfer Time

There' no definition of "Ideal TCP Transfer Time", but I conclude from
the paragraph below that it is the theoretical raw capacity combined
with a block of information that needs to be transferred.  Since there's
no flow control/ARQ at all, I would be more comfortable calling the
denominator "Ideal Transfer Time", or "Theoretical Transfer Time".

It would also seem appropriate to explain to the user that no TCP
will ever achieve the same performance as Theoretical, and perhaps
this means we are comparing apples and oranges in this metric...


>    An example would be the bulk transfer of 100 MB upon 5 simultaneous
>    TCP connections over a 500 Mbit/s Ethernet service

Is this the Committed Information Rate (CIR) of the Ethernet Service?

>                                                      (each connection
>    uploading 100 MB).  Each connection may achieve different throughputs
>    during a test and the overall throughput rate is not always easy to
>    determine (especially as the number of connections increases).

----------------------------------------------------------------------------

3. TCP Throughput Testing Methodology
...
>    2. Baseline

(Average?)

>Round-trip Delay and

Raw capacity (without TCP flow control).

>      . These measurements
>    provide estimates of the ideal TCP window size, which will be used in
>    subsequent test steps.

-----------------------------------------------------------------------------
(later, S3)

>    - TCP implementation used by the test host

and the OS version number,

>  i.e. Linux OS kernel

2.6.......

>    using TCP Reno, TCP options supported, etc.  This will obviously be
>    more important when using custom test equipment where the TCP
>    implementation may be customized or tuned to run in higher
>    performance hardware
>
>
>    - Most importantly, the TCP test host must be capable of generating
>    and receiving stateful TCP test traffic at the full link speed of the
>    network under test. As a general rule of thumb, testing TCP
>    throughput at rates greater than 100 Mbit/sec generally requires high
>    performance server hardware or dedicated hardware based test tools.

(add)
     Thus, other devices cannot realize higher TCP throughput, and user
     expectations SHOULD be set accordingly with user manual or notes on
     the results report.


>    - To measure RTT and TCP Efficiency per connection, this will



At 10:13 AM 9/10/2010, Al Morton wrote:
>Barry, Rudi,
>
>My recollection from the March IPPM session is that
>  - we agreed to take this up as an Informational RFC
>  - we aren't defining metrics here
>   (if we were, it would be standards track)
>
>So, I agree with the "Framework" title change.
>I would clarify that "Type-P" metric specifications
>are un-attempted with this scope (what the draft does now).
>
>I will hopefully add some more comments later today (sigh).
>
>regards,
>Al
>
>At 09:44 AM 9/10/2010, Barry Constantine wrote:
>>...
>>I thought we agreed that if this were to be retitled a framework 
>>(and informational versus a standard), then the Type P metrics 
>>could remain unaddressed.
>>
>>Please let me know if I am misunderstanding, will definitely 
>>incorporate your comments in this email (minus the Type P metric work).
>>
>>
>>Thanks,
>>
>>Barry
>>
>>