Re: [ippm] Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-method-06: (with DISCUSS)

[Magnus Westerlund <[e-mail]>]

Hi Al,

Please see inline. 

On Fri, 2021-02-26 at 06:05 +0000, MORTON, ALFRED C (AL) wrote:
> Hi Magnus, Thanks for your review.
> 
> Please see replies to your review and comments from Len and me, consolidated
> and marked [acm] below,
> 
> Al
> 
> > -----Original Message-----
> > From: Magnus Westerlund via Datatracker [mailto:[e-mail]]
> > Sent: Thursday, February 25, 2021 9:19 AM
> > To: The IESG <[e-mail]>
> > Cc: [e-mail]; [e-mail];
> > [e-mail]; Ian Swett <[e-mail]>; [e-mail];
> > [e-mail]
> > Subject: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-
> > method-06: (with DISCUSS)
> > 
> > Magnus Westerlund has entered the following ballot position for
> > draft-ietf-ippm-capacity-metric-method-06: Discuss
> > 
> 
> ...
> > 
> > ----------------------------------------------------------------------
> > DISCUSS:
> > ----------------------------------------------------------------------
> > 
> > A) Section 8. Method of Measurement
> > 
> > I think the metrics are fine, what makes me quite worried here is the
> > measurement method. My concerns with it are the following.
> > 
> > 1. The application of this measurement method is not clearly scoped.
> > Therefore
> > I will assume that across the Internet measurements are possible. 
> 
> [acm]
> 
> Like all the ad hoc Internet Speed measurements in the world today, our
> primary interest is access measurement.  I added "Applicability" to the Scope
> and Goals section 2, as follows:
> 
> NEW paragraph
> The primary application of this metric and method of measurement described
> here is the same as in Section 2 of RFC7479, where:
> 
> o The access portion of the network is the focus of this problem statement.
> The user typically subscribes to a service with bidirectional access partly
> described by rates in bits per second.
> 

Good, I guess that this was the case, but it wasn't stated explicitly. 

The reason I am worried is that this metric and also the method of measurement
as noted creates impact on the network. What makes me extra worried is that the
congestion avoidance property is currently not well documented and the rather
big parameterization space that is left without guidance and will impact the
properties. 

> -=-=-=-=-=-=-=-
> 
> [acm] Obviously we won't be able to stop application beyond access, just like
> we can't stop people from grabbing one of any number of utilities and anywhere
> and sending at any rate they can achieve, but we have made an applicability
> statement. 
> 
> > However in
> > that context I think the definition and protection against severe congestion
> > has significant short comings. The main reason is that the during a
> > configurable time period (default 1 s) the sender will attempt to send at
> > a specified rate by a table independently on what happens during that
> > second.
> 
> [acm] 
> Not quite, 1 second is the default measurement interval for Capacity, but
> sender rate adjustments occur much faster (and we add a default at 50ms). This
> is a an important point (and one that Ben also noticed, regarding variable F
> in section 8.1). So, I have added FT as a parameter in section 4:
> 
> o FT, the feedback time interval between status feedback messages
> communicating measurement results, sent from the receiver to control the
> sender. The results are evaluated to determine how to adjust the current
> offered load rate at the sender (default 50ms)
> 
> -=-=-=-=-=-=-=-
> Note that variable F in section 8.1 is redundant with parameter F in Section
> 4, 
> the number of flows (in-06). So we changed the section 8.1 variable F to FT 
> in the working text.

Okay, that makes things clearer. With all the equal intervals in the metrics I
had missinterpreted that also the transmission would be uniform during the
measurement intervals. 

However, when rereading Section 8.1 I do have to wonder if the non-cumaltive
feedback actually creates two issues. First, it appears to loose information for
reordering that crosses the time when the FT timer fires, due to reset. In
addition if the feedback is not reliable it looses the information for that
interval. And making feedback reliabel could cause worse HOL issues for reacting
to later feedback that are recived prior to the lost one. 


> 
> 
> > 
> > 2. The algorithm for adjusting rate is table driven but give no guidance on
> > how
> > to construct the table and limitations on value changes in the table. In
> > addition the algorithm discusses larger steps in the table without any
> > reflection of what theses steps sides may represent in offered load.
> 
> [acm] 
> We can add (Len suggested the following text addition):
> OLD
> 8.1. Load Rate Adjustment Algorithm
> 
> A table SHALL be pre-built defining all the offered load rates that
> will be supported (R1 through Rn, in ascending order, corresponding
> to indexed rows in the table). Each rate is defined as datagrams of...
> 
> NEW
> 8.1. Load Rate Adjustment Algorithm
> 
> A table SHALL be pre-built defining all the offered load rates that
> will be supported (R1 through Rn, in ascending order, corresponding
> to indexed rows in the table). It is RECOMMENDED that rates begin with
> 0.5 Mbps at index zero, use 1 Mbps at index one, and then continue in
> 1 Mbps increments to 1 Gbps. Above 1 Gbps, and up to 10 Gbps, it is
> RECOMMENDED that 100 Mbps increments be used. Above 10 Gbps,
> increments of 1 Gbps are RECOMMENDED. Each rate is defined as...

Is this what you actually used in your test implementation? At my first glance
this recommendation looks to suffer from rather severe step effects and also
make the respond to losses behave strange around the transitions. Wouldn't some
type of logarithmic progression be more appropriate here for initial probing?

If I have 1 GBPS line rate, there is a 1000 steps in the table to this value.
Even if I increase with the suggested 10 steps until first congestion seen, it
will take 100 steps, and with 50 ms feedback interval that is 5 seconds before
it is in the right ball park. And I get one random loss at 10 mbps, then its 990
steps, In such a situation the whole measurement period (10 s) would be over
before one has reached actual capacity. 

To me it appears that the probing (slow start) equivalent do need logarithmic
increase to reach likely capacity quickly. Then how big the adjustment is
actually dependent on what extra delay one consider the target for the test.
Having a step size of 1 GBPS if probing a 2.5 GBPS path would likely make it
very hard to keep the delay in the intended interval when it would fluxtuate
between 500 mbps to much traffic and then 500 mbps to little. Sure with
sufficiently short FT it will likely work in this algorithm. However, I wonder
about regulation stability here for differnet RTT, FTs and buffer depth
fluxtuations. 

From my perspective I think this is a indication that the load rate adjustment
algorithm is not ready to be a standards track specification. 

I would recommend that you actually take out the control algorithm and write a
high level functional description of what needs to happen when measuring this
capacity. 

If I understand this correctly the requirements on the measurement are the
following.

- Need to seek the available capacity so that several measurement period are
likely to be done at capacity
- Must not create persistent congestion as the capacity measurement should be
based on traffic capacity that doesn't cause more standing queue that X, where X
is some additiona delay in ms compared to minimal one way delay. And X is
actually something that is configurable for a measurement campagin as capacity
for a given one way delay and delay variation can be highly relevant to know. 

What else is needed?

Are synchronized clocks needed or just relative delay changes necessary? 

> 
> -=-=-=-=-=-=-=-
> 
> > 
> > 3. Third the algorithms reaction to any sequence number gaps is dependent on
> > delay and how it is related to unspecified delay thresholds. Also no text
> > discussion how these thresholds should be configured for safe operation.
> 
> [acm] 
> We can add some details in the paragraph below:
> OLD
> If the feedback indicates that sequence number anomalies were detected OR 
> the delay range was above the upper threshold, the offered load rate is
> decreased. 
> Also, if congestion is now ...
> NEW
> If the feedback indicates that sequence number anomalies were detected OR 
> the delay range was above the upper threshold, the offered load rate is
> decreased. 
> The RECOMMENDED values are 0 for sequence number gaps and 30-90 ms for lower 
> and upper delay thresholds. Also, if congestion is now ...

Ok, but the delay values as I noted before highly dependent of what my goal with
the capacity metric is. If I want to figure out the capacity for like say XR or
cloud gaming applications that maybe have much lower OWD variances and absolute
values so maybe my values are 10-25 ms. 

How much explaration have you done of the control stability over a range of
parameters? Do you have any material about that? 


> 
> -=-=-=-=-=-=-=-
> 
> Please also note many requirements for safe operation in Section 10, 
> Security Considerations.
> 
> > 
> > B) Section 8. Method of Measurement
> > 
> > There are no specification of the measurement protocol here that provides
> > sequence numbers, and the feedback channel as well as the control channel.
> 
> [acm] 
> That is correct. The Scope does not include protocol development.
> 
> > Is this intended to use TWAMP?
> 
> [acm] 
> Maybe, but a lot of extensions would be involved.



> 
> > 
> > From my perspective this document defines the metrics on standards track
> > level. However, the method for actually running the measurements are not
> > specified on a standards track level. 
> 
> [acm]
> In IPPM work, the methods of measurement are described more broadly than 
> the metrics, as actions and operations the Src and Dst hosts perform to 
> send and receive, and calculate the results. 
> 
> IPPM Methods of Measurement have not included protocol requirements in 
> the past, in any of our Standards Track Metrics RFCs.  In fact, we developed 
> a measurement-specific criteria for moving our RFCs along the standards track 
> that has nothing to do with protocols or interoperability. 
> See BCP 176 aka RFC 6576:  https://tools.ietf.org/html/rfc6576
> IP Performance Metrics (IPPM) Standard Advancement Testing 
> 
> > No one can build implementation. 
> 
> [acm] 
> I'm sorry, but that is not correct.  Please see Section 8.4.

Sorry, that was poorly formulated. I mean that you can't give this specification
to a guy on a island without external communication and have them implemented it
and it will work with someone else implementation. You have clearly implemented
a solutiont that works for some set of parameters. And I am asking how much of
the reasonable parameter space you have tested. 

Based on this discussion I don't think I can build an implementation that
fulfills the measurement goals, becasue I have questions about them. And I
suspect it would take substantial amount of experimentation to get it to work
correctly over a broader range of input parameters. 



> 
> > And if the section is
> > intended to provide requirements on a protocol that performs these
> > measurements
> > I think several aspects are missing. There appear several ways forward here
> > to
> > resolve this; one is to split out the method of measurement and define it
> > separately to standard tracks level using a particular protocol, another
> > is to write it purely as requirements on a measurement protocols.
> 
> [acm] 
> As stated above, connecting a method with a single protocol is not IPPM's way.

That is fine. However, I find the attempt to specify a specific load regulator
in the method of measurement to take this specification beyond a general method
of measurment. The high level requirement appear to be that to correctly find
the capacity, and that requires that one load to the point where buffers are
filled sufficiently to introduce extra delay or where AQM starts dropping or
marking some of the load. Thus, I am questioning if the described algorithm will
adqueately solve that issue over a wider range of parameters. 

So if you have more information to show at least which range it has been proven
to do its work and with what input parameters? I hope you understand that I
expect this load control algorithm to get simularly scrutinized to congestion
control algorithms that we standardize in IETF. 

I would very much prefer to take out the load algorithm and place it in a
seperate document where it can have a tighter scope description and more
discussion about about that it does its job. 

I hope this clarifies what my concerns are with this document in its current
form. 

Cheers

Magnus