Re: [ippm] Benjamin Kaduk's No Objection on draft-ietf-ippm-capacity-metric-method-06: (with COMMENT)

"MORTON, ALFRED C (AL)" <acm@research.att.com> Thu, 25 February 2021 03:08 UTC

Return-Path: <acm@research.att.com>
X-Original-To: ippm@ietfa.amsl.com
Delivered-To: ippm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 26C7C3A0D76; Wed, 24 Feb 2021 19:08:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level:
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YGue1h7PEb9d; Wed, 24 Feb 2021 19:08:40 -0800 (PST)
Received: from mx0a-00191d01.pphosted.com (mx0a-00191d01.pphosted.com [67.231.149.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 159473A0D75; Wed, 24 Feb 2021 19:08:39 -0800 (PST)
Received: from pps.filterd (m0049287.ppops.net [127.0.0.1]) by m0049287.ppops.net-00191d01. (8.16.0.43/8.16.0.43) with SMTP id 11P35eB8035975; Wed, 24 Feb 2021 22:08:33 -0500
Received: from tlpd255.enaf.dadc.sbc.com (sbcsmtp3.sbc.com [144.160.112.28]) by m0049287.ppops.net-00191d01. with ESMTP id 36w9tm93hw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 24 Feb 2021 22:08:32 -0500
Received: from enaf.dadc.sbc.com (localhost [127.0.0.1]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id 11P38UR2032787; Wed, 24 Feb 2021 21:08:30 -0600
Received: from zlp30497.vci.att.com (zlp30497.vci.att.com [135.46.181.156]) by tlpd255.enaf.dadc.sbc.com (8.14.5/8.14.5) with ESMTP id 11P38QKj032730 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 24 Feb 2021 21:08:26 -0600
Received: from zlp30497.vci.att.com (zlp30497.vci.att.com [127.0.0.1]) by zlp30497.vci.att.com (Service) with ESMTP id 373954016998; Thu, 25 Feb 2021 03:08:26 +0000 (GMT)
Received: from clph811.sldc.sbc.com (unknown [135.41.107.12]) by zlp30497.vci.att.com (Service) with ESMTP id DF1124016997; Thu, 25 Feb 2021 03:08:25 +0000 (GMT)
Received: from sldc.sbc.com (localhost [127.0.0.1]) by clph811.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id 11P38PVt047633; Wed, 24 Feb 2021 21:08:25 -0600
Received: from mail-green.research.att.com (mail-green.research.att.com [135.207.255.15]) by clph811.sldc.sbc.com (8.14.5/8.14.5) with ESMTP id 11P38Dql046619; Wed, 24 Feb 2021 21:08:13 -0600
Received: from exchange.research.att.com (njbdcas1.research.att.com [135.197.255.61]) by mail-green.research.att.com (Postfix) with ESMTP id BA2C210A18C3; Wed, 24 Feb 2021 22:08:12 -0500 (EST)
Received: from njmtexg5.research.att.com ([fe80::b09c:ff13:4487:78b6]) by njbdcas1.research.att.com ([fe80::8c6b:4b77:618f:9a01%11]) with mapi id 14.03.0468.000; Wed, 24 Feb 2021 22:08:29 -0500
From: "MORTON, ALFRED C (AL)" <acm@research.att.com>
To: Benjamin Kaduk <kaduk@mit.edu>, The IESG <iesg@ietf.org>
CC: "draft-ietf-ippm-capacity-metric-method@ietf.org" <draft-ietf-ippm-capacity-metric-method@ietf.org>, "ippm-chairs@ietf.org" <ippm-chairs@ietf.org>, "ippm@ietf.org" <ippm@ietf.org>, Ian Swett <ianswett@google.com>, "tpauly@apple.com" <tpauly@apple.com>
Thread-Topic: Benjamin Kaduk's No Objection on draft-ietf-ippm-capacity-metric-method-06: (with COMMENT)
Thread-Index: AQHXCubpIAgrxqGmcU+aUTtf1usNw6pnzooA
Date: Thu, 25 Feb 2021 03:08:29 +0000
Message-ID: <4D7F4AD313D3FC43A053B309F97543CF01476A0549@njmtexg5.research.att.com>
References: <161419645471.18083.16706266293896961774@ietfa.amsl.com>
In-Reply-To: <161419645471.18083.16706266293896961774@ietfa.amsl.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [24.148.42.167]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369, 18.0.761 definitions=2021-02-25_01:2021-02-24, 2021-02-25 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_policy_notspam policy=outbound_policy score=0 adultscore=0 phishscore=0 impostorscore=0 mlxscore=0 malwarescore=0 spamscore=0 priorityscore=1501 bulkscore=0 lowpriorityscore=0 mlxlogscore=999 clxscore=1011 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102250025
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/p5oxuxcOhpBMDCnXQOPR2tNwHjo>
Subject: Re: [ippm] Benjamin Kaduk's No Objection on draft-ietf-ippm-capacity-metric-method-06: (with COMMENT)
X-BeenThere: ippm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ippm>, <mailto:ippm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm/>
List-Post: <mailto:ippm@ietf.org>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ippm>, <mailto:ippm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 25 Feb 2021 03:08:43 -0000

Hi Ben,

Thanks for your detailed review; it will be a better draft when we're done, as always!

All the changes identified below are implemented in my working version.

Please see replies below, [acm]
Al

> -----Original Message-----
> From: Benjamin Kaduk via Datatracker [mailto:noreply@ietf.org]
> Sent: Wednesday, February 24, 2021 2:54 PM
> To: The IESG <iesg@ietf.org>
> Cc: draft-ietf-ippm-capacity-metric-method@ietf.org; ippm-chairs@ietf.org;
> ippm@ietf.org; Ian Swett <ianswett@google.com>om>; tpauly@apple.com;
> tpauly@apple.com
> Subject: Benjamin Kaduk's No Objection on draft-ietf-ippm-capacity-metric-
> method-06: (with COMMENT)
> 
> Benjamin Kaduk has entered the following ballot position for
> draft-ietf-ippm-capacity-metric-method-06: No Objection
> 
...
> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> Section 3
> 
>    5.  Less emphasis on ISP gateway measurements, possibly due to less
>        traffic crossing ISP gateways in future.
> 
> nit: sentence fragment.
[acm] 
ok
s/Less/There is less/

> 
> Section 5
> 
>    This section sets requirements for the following components to
>    support the Maximum IP-layer Capacity Metric.
> 
> editorial/nit: I don't think I understand what "the following
> components" are. 
[acm] 
I see your point. In the IPPM Framework RFC2330 (Section 11), the singleton and the sample metrics are defined separately, and statistics are the metrics that summarize samples. 

We could say this more directly:

This section sets requirements for the singleton metric that supports the Maximum IP-layer Capacity Metric definition in Section 6.


> Is this referencing some preexisting template for a
> metric definition?  (Same for §6 and §7.)
[acm]
Yes.  The original template has its origins in RFC1242 (1991 in BMWG).  BMWG and IPPM shared a single session at IETF meetings during IPPM's formation, and the first IPPM metric in RFC2498 uses a template very similar to RFC1242.

Sorry if this is TMI.

> 
> Section 5.3
> 
>    The number of these IP-layer bits is designated n0[dtn,dtn+1] for a
>    specific dt.
> 
> (editorial) I'm a little confused by this notation for n0.  In Section 4
> we say that "dtn" references a specific sub-interval, but the brackets
> and two items look like they should be the start an end of an interval,
> for which perhaps just the 'n' would be more appropriate (but we'd want
> a half-open interval, and would need to worry about 0- vs 1-indexing for
> n, etc.).
[acm] 
Maybe if we tweak the definition of dtn in section 4, we can help without causing more confusion:
OLD
o  dtn, a specific sub-interval, n, one of m sub-intervals in I
NEW
o  dtn, the beginning boundary of a specific sub-interval, n, one of m sub-intervals in I


> 
>    Anticipating a Sample of Singletons, the interval dt SHOULD be set to
>    a natural number m so that T+I = T + m*dt with dtn+1 - dtn = dt and
>    with 1 <= n <= m.
> 
> nit: "dt [...] set to m" doesn't make any sense.
> nit: this looks like more of a "for 1 <= n <= m" than an "and with".
[acm] 
Ok, here's the revised text I suggest:

Anticipating a Sample of Singletons, the number of sub-intervals with duration dt SHOULD be set to a natural number m, so that T+I = T + m*dt with dtn+1 - dtn = dt for 1 <= n <= m.

(The definition of dt in Section 4 effectively defines m with fewer dependencies, but this sentence tries to tie the whole system of variables together.)

> 
>    Mathematically, this definition can be represented as:
> 
>                                       ( n0[dtn,dtn+1] )
>                       C(T,dt,PM) = -------------------------
>                                              dt
> 
> (nit) the "n" appears (in 'dtn') only on the right side but there is no
> operation applied over it, so this "equation" seems unbalanced without
> also specifying 'n'.  I think the introductory text should mention
> something about "for each n" or for a given interval dtn" or similar.
[acm] 
Thanks, that's a good fix.

> 
>    o  n0 is the total number of IP-layer header and payload bits that
>       can be transmitted in Standard Formed packets [RFC8468] from the
> 
> (nit) RFC 8468 spells it "standard-formed".
[acm] 
Thanks, fixed.

> 
>    o  C(T,dt,PM) the IP-Layer Capacity, corresponds to the value of n0
>       measured in any sub-interval ending at dtn (meaning T + n*dt),
>       divided by the length of sub-interval, dt.
> 
> If it's supposed to be "ending at dtn", then why is the "dtn+1" in the
> picture at all?
[acm] 
Good catch, there was a version where dtn was at the end of the sub-interval in the IPPM memo, but we moved it for consistency with other specifications.
Fixed as "...beginning at dtn, ..."
Section 6.3 too.

> 
>    o  PM represents other performance metrics [see section 5.4 below];
>       their measurement results SHALL be collected during measurement of
>       IP-layer Capacity and associated with the corresponding dtn for
>       further evaluation and reporting.
> 
> (nit) this seems to duplicate (be a subset of) the paragraph before
> "Mathematically, this definition can be represented".
[acm] 
It's a simplification of the paragraph above: it seemed odd not to list PM and a short explanation along with the other dependent aspects.
 
> 
>    o  The bit rate of the physical interface of the measurement device
>       must be higher than that of the link whose C(T,I,PM) is to be
>       measured.
> 
> (nit) I thought we were measuring a path, not a link.
[acm] 
yes, path, and s/device/devices/
Similar in 6.3.


> 
> Section 5.4
> 
>    RTD[dtn-1,dtn] is defined as a sample of the [RFC2681] Round-trip
> 
> Do we really want to be using n0[dtn,dtn+1] but RTD[dtn-1,dtn]?  Can we
> pick a consistent sub-interval notation?
[acm] 
Sorry to have left a few of these behind in the notation conversion!
Fixed.

> 
> Section 6.3
> 
>    Define the Maximum IP-layer capacity, Maximum_C(T,I,PM), to be the
>    maximum number of IP-layer bits n0[dtn,dtn+1] that can be transmitted
>    in packets from the Src host and correctly received by the Dst host,
> 
> (nit) the relevant formulae include a dt divisor, but I don't see
> anything in this prose that would correspond to such a divisor.
[acm] 
Yes, you're right:
Define the Maximum IP-layer capacity, Maximum_C(T,I,PM), to be the 
maximum number of IP-layer bits n0[dtn,dtn+1] divided by dt that ...


> 
>    The interval dt SHOULD be set to a natural number m so that T+I = T +
>    m*dt with dtn+1 - dtn = dt and with 1 <= n <= m.
> 
> nit: "dt [...] set to m" doesn't make any sense.
> nit: this looks like more of a "for 1 <= n <= m" than an "and with".
[acm] 
Same fix as Section 5.3

The number of sub-intervals with duration dt SHOULD be set to a natural number m, so that T+I = T + m*dt with dtn+1 - dtn = dt for 1 <= n <= m.

> 
>    Mathematically, this definition can be represented as:
> 
>                                       max  ( n0[dtn,dtn+1] )
>                                      [T,T+I]
>                 Maximum_C(T,I,PM) = -------------------------
>                                                dt
>                where:
>                   T                                      T+I
>                   _________________________________________
>                   |   |   |   |   |   |   |   |   |   |   |
>               dtn=1   2   3   4   5   6   7   8   9  10  n+1
>                                                      n=m
> 
> (nit) as mentioned previously, the definition of "dtn" lists it as being
> the sub-interval, not the boundary point/time of a sub-interval.
[acm] 
Right, tweaked in the parameter definition now, section 4.


> 
> Section 6.5
> 
>    If traffic conditioning (e.g., shaping, policing) applies along a
>    path for which Maximum_C(T,I,PM) is to be determined, different
>    values for dt SHOULD be picked and measurements be executed during
>    multiple intervals [T, T+I].  A single constant interval dt SHOULD be
>    chosen so that is an integer multiple of increasing values k times
>    serialisation delay of a path MTU at the physical interface speed
>    where traffic conditioning is expected.  [...]
> 
> nit: "so that is an integer multiple" seems to be missing a word.
[acm] 
Yes, "it" is the word, thanks.

> 
> Also, this seems to say that different values for dt SHOULD be picked,
> but also that a constant dt SHOULD be chosen.  How can those both be
> recommended and be consistent with each other?  (I mean, I assume that
> the intent is to multiple runs with different (fixed) dt, but that's not
> what the text seems to say.)
[acm] 
Yes, I see your point, this paragraph describes tests at multiple values of dt.

OLD
A single constant interval dt SHOULD be chosen so that it is an integer 
multiple of increasing values k times serialisation delay of a path MTU 
at the physical interface speed where traffic conditioning is expected.
NEW
Each duration dt SHOULD be chosen so that it is an integer 
multiple of increasing values k times serialization delay of a path MTU 
at the physical interface speed where traffic conditioning is expected.


> 
> Section 7.3
> 
>    Define the IP-layer Sender Bit Rate, B(S,st), to be the number of IP-
>    layer bits (including header and data fields) that are transmitted
>    from the Source during one contiguous sub-interval, st, during the
>    test interval S (where S SHALL be longer than I), and where the
>    fixed-size packet count during that single sub-interval st also
>    provides the number of IP-layer bits in any interval: n0[stn-1,stn].
> 
> (1) there doesn't seem to be any restriction that the observed packets
> list Dst as the destination address, so formally it seems this would
> count *all* traffic generated by Sender, not just the traffic relevant
> for the (path capacity) test.
[acm] 
Yes, I agree that's not what we intended. Let's add both Src and Dst:

... that are transmitted from the Source with address pair Src and Dst 
during one contiguous sub-interval, st, ...

> (2) It seems a little unfortunate that we reuse the 'n0' symbol here for
> a different meaning than in the earlier capacity metrics.
[acm] 
Yes, the earlier definition of n0 demands correct reception at the destination
and we can't assure that at a sender measurement point with UDP.

I think we can just drop n0:
OLD
... and where the fixed-size packet count during that single sub-interval st also 
provides the number of IP-layer bits in any interval: n0[stn-1,stn].
NEW
... and where the fixed-size packet count during that single sub-interval st also 
provides the number of IP-layer bits in any interval, [stn,stn+1].

Now, it strikes me that we didn't define stn with the same rigor as dtn, so I added 
a definition for stn in section 7.2 Parameters section above:

o  stn, the beginning boundary of a specific sub-interval, n, one of N sub-intervals in S



> 
>    Measurements according to these definitions SHALL use the UDP
>    transport layer.  Any feedback from Dst host to Src host received by
>    Src host during an interval [stn-1,stn] MUST NOT result in an
>    adaptation of the Src host traffic conditioning during this interval
>    (rate adjustment occurs on st boundaries).
> 
> Hmm, this "MUST NOT" is interesting, as it seems to imply extremely
> tight coordination between the measurement point for this metric and the
> Source itself.  (Note that the toplevel §7 admits the possibility that
> measurement will occur at a location other than the Src host to network
> path interface, via "(or as close as practical)".)
[acm] 
Yes, that's what we had in mind, but we also envision the measurement point
*internal* to the Src host making synchronization of rate adjustments and 
measurements more practical, too. We're just asking to get very close to the 
physical interface.

so in toplevel §7:
... (or as close as practical within the Src host).

But now, all the realities of measurement in front of mind, I think SHOULD NOT
will be sufficient.


> 
> Section 8.1
> 
>    At the beginning of a test, the sender begins sending at rate R1 and
>    the receiver starts a feedback timer at interval F (while awaiting
> 
> It's a little hard to search for, but I didn't find any previous mention
> of 'F' or it being defined as a parameter or term.  Should it be a
> listed parameter somewhere?
[acm] 
We define a lot of variables in this section, with limited scope of use.
F is one of them, but I found that we had already defined F in Section 4!
So... F becomes FT, and like R1 and ss and cc, gets no special treatment
beyond a definition in the text (IF you're ok with that).

> 
>    If the feedback indicates that sequence number anomalies were
>    detected OR the delay range was above the upper threshold, the
>    offered load rate is decreased.  Also, if congestion is now confirmed
>    by the current feedback message being processed, then the offered
>    load rate is decreased by more than one rate (e.g., Rx-30).  [...]
> 
> Does "congestion is now confirmed" mean that "congestion confirmed" is
> like a one-way latch and this transition only occurs at most once over
> the course of a test?  Or could the Rx-30 happen multiple times?
> (The pseudocode indicates the former.)
[acm] 
Yes, we are trying to describe the pseudocode, and "congestion confirmed"
latches when the slowAdjCount equals the slowAdjThresh (and after that,
slowAdjCount continues upward, the slowAdjCount < slowAdjThresh condition
fails and slowAdjCount is never reset to zero).

So, I think we want to say:
OLD
Also, if congestion is now confirmed by the current feedback message...
NEW
Also, if congestion is now confirmed for the first time by the 
current feedback message...

> 
>    If the feedback indicates that there were no sequence number
>    anomalies AND the delay range was above the lower threshold, but
>    below the upper threshold, the offered load rate is not changed.
> 
> The way this is written suggests that there will always be a lower and
> an upper threshold for delay, but the rest of the document so far didn't
> give me that impression.  E.g., we talk about PM only as "at least one
> fundamental metric and target performance threshold MUST be supplied",
> and to me having both upper and lower thresholds would be two
> thresholds, not one.
[acm] 
That's true, we tried not to force our current/best algorithm into the 
metric definition. We require some measurement to use for feedback 
in rate adjustment, otherwise you just have iPerf or other fixed rate 
tools that can only blast packets.

You can build an "ok" feedback system with just one metric and one threshold, 
but there are drawbacks and it may take longer duration tests to measure
the true maximum capacity due to a technical limitation.
So, we gave a really good algorithm, in 8.1, in pseudocode, and even 
in running code


> 
> Section 8.2
> 
>    Here, as with any Active Capacity test, the test duration must be
>    kept short. 10 second tests for each direction of transmission are
>    common today.  The default measurement interval specified here is I =
>    10 seconds).  In combination with a fast search method and user-
>    network coordination, the concerns raised in RFC 6815[RFC6815] are
>    alleviated.  [...]
> 
> I skimmed RFC 6815 and had a bit of a hard time making the connection
> for why combining a 10-second interval, fast search method, and
> user-network coordination alleviate the concerns of RFC 6815.  There
> doesn't seem to be much in 6815 itself about how testing in production
> can be done safely,
[acm] 
That's certainly true, but we did say:

   The world will not spin off axis while waiting for appropriate and
   standardized methods to emerge from the consensus process.

> so my current working assumption is that the
> conclusion presented here reflects the results of "new work" being
> recorded for the first time (in the RFC series) in this document.  
[acm] 

When you put it that way, yes. Although it is a different metric from
RFC2544 Throughput, the load adjustment search algorithm alone helps
to make this method safer to use than any fixed-rate UDP packet blaster,
or even a binary search-controlled measurement because of near real-time 
feedback.

The other reasons why this work is different are that
RFC 2544 Throughput measurements intend to overload the isolated test 
environment for extended periods of time: 

24. Trial duration (from https://tools.ietf.org/html/rfc2544#section-24)

   The aim of these tests is to determine the rate continuously
   supportable by the DUT.  The actual duration of the test trials must
   be a compromise between this aim and the duration of the benchmarking
   test suite.  The duration of the test portion of each trial SHOULD be
   at least 60 seconds. ...

Many automated RFC2544 test devices start a test at the highest load, and
search their way down to the zero-loss Throughput, subjecting the 
device under test to potentially extreme overload multiple times before 
reaching the test outcome

> If that assumption is correct, I'd suggest spending some more words to
> support the conclusion, e.g., making analogies to other "normal" traffic
> patterns and how the benchmarking setup is not qualitatively different
> from them.
[acm] 

OK, I put some more background together and made the case stronger:
the memo we wrote hear is exactly what the RFC6815 authors were asking for.

The Max IP Capacity metric and method for assessing is very different from classic RFC2544
Throughput metric and methods : it uses near-real-time load adjustments that are sensitive to loss and delay, similar to other congestion control algorithms used on the Internet every day, along with limited duration. On the other hand, RFC2544 Throughput measurements can produce sustained overload conditions for extended periods of time. Individual trials in a test governed by a binary search can last 60 seconds for each step, and the final confirmation trial may be even longer. This is very different from "normal" traffic levels, but overload conditions are not a concern in the isolated test environment. The concerns raised in RFC6815 were that RFC2544 methods would be let loose on production networks, and instead the authors challenged the standards community to develop metrics and methods like those described in this memo.


> 
> Section 8.3
> 
>    As testing continues, implementers should expect some evolution in
>    the methods.  The ITU-T has published a Supplement (60) to the
>    Y-series of Recommendations, "Interpreting ITU-T Y.1540 maximum IP-
>    layer capacity measurements", [Y.Sup60], which is the result of
>    continued testing with the metric and method described here.
> 
> I pulled up the [Y.Sup60] reference, and it does not seem to reference
> this draft by name.  On what basis do we conclude that it "is the result
> of continued testing with the metric and method described here"?
> Skimming/searching, I do see many similar formulae and methods
> presented, but how do we conclude they are precisely the same?
[acm] 
I'll soften that a bit. The Max IP-Layer Capacity metric is 
the same, but it is likely that a few details in the method have diverged
over time -- much of the ITU-T testing and spec development came first.

NEW
... [Y.Sup60], which is the result of continued testing with the metric, 
and those results have improved the method described here.



> 
> Section 10
> 
> Should we say something about making sure that I is reasonably bounded?
> IIRC we say so elsewhere in the text but not exactly here.
[acm] 
I added a direct reference to I at the end of item 6.:

... Testing with the Service Provider's measurement hosts SHOULD be limited in frequency and/or overall volume of test traffic (for example, the range of I duration values SHOULD be limited).
> 
>    2.  A REQUIRED user client-initiated setup handshake between
>        cooperating hosts and allows firewalls to control inbound
>        unsolicited UDP which either go to a control port [expected and
>        w/authentication] or to ephemeral ports that are only created as
>        needed.  [...]
> 
> nit: the grammar is odd in the first part of this sentence; the part
> before the "and" doesn't seem like it can join up with anything after
> the "and".  Is the intent something like "It is REQUIRED to have a user
> client-initiated setup handshake between cooperating hosts that allows
> firewalls to [...]"?
[acm] 
Thanks, good re-wording, it's in.

> 
>    3.  Integrity protection for feedback messages conveying measurements
>        is RECOMMENDED.
> 
> (In some sense you want authentication as well as integrity protection.)
[acm] 
Yes. The running code has optional authentication now.

NEW
3. Client-server authentication and integrity protection for feedback 
   messages conveying measurements is RECOMMENDED.

> 
>    5.  Senders MUST be rate-limited.  This can be accomplished using the
>        pre-built table defining all the offered load rates that will be
>        supported (Section 8.1).  The recommended load-control search
>        algorithm results in "ramp up" from the lowest rate in the table.
> 
> nit: since (effectively) each implementation will have their own
> pre-built table, I think it should be "using a pre-built table".
[acm] 
OK, "a" it is.


> 
> Appendix 13
> 
> If we start at Rx (row) 1, is it going to cause problems when we drop
> down to Rx = 0 in the loss/congestion cases?
[acm] 
It would, but the current table includes a Row zero, which is where we 
cold-start. I guess it would less confusing to say:

Rx = 0  # The current sending rate (equivalent to a row of the table)


> 
> The mechcanism in the pseudocode to stop taking large increments in
> sending rate above the "hSpeedThresh" does not seem to be described in
> the prose in §8.1.  (That said, it seems like a good idea, given the
> likely table composition.)
[acm] 
It's getting late here now, but I think it's in the first two If statements:

if ( seqErr == 0 && delay < lowThresh ) {                         # no loss or delay problems, and
        if ( Rx < hSpeedTresh && slowAdjCount < slowAdjThresh ) { # Rate < hSpeedThresh && etc.
                        Rx += highSpeedDelta;                     # can still use large increments
                        slowAdjCount = 0;
        } else {                                                  # otherwise (Rate >= hSpeedThresh)
                        if ( Rx < maxLoadRates - 1 )              # after checking headroom,
                                        Rx++;                     # can only increase by one


> 
> (Also, indenting one tab for the outer conditionals and two more for the
> inner ones looks a bit unusual.)
[acm] 
I like unusual :-)

> 
> Section 14
> 
> It's not entirely clear to me why RFC 2330 is classified as normative
> but RFC 7312 is informative, just based on the locations where they are
> referenced.
[acm] 
It's more than that. RFC 7312 describes some unusual access conditions that might be encountered and is only cited at the end of the Intro, once. Certainly the measured evidence of bimodal (turbo-mode) access behavior is in the category of RFC 7312 "messy stuff", but we still manage that pretty well and it's not specifically mentioned in 7312.

OTOH, RFC 2330 is the IPPM Framework, with an exception to be Informative status, yet Normative in our memos, and much is owed to 2330, starting with the singleton, sample, statistic metric development, unmentioned stuff about clocks and accuracy, etc.
> 
>