Re: [alto] Lars Eggert's Discuss on draft-ietf-alto-performance-metrics-19: (with DISCUSS and COMMENT)

["Y. Richard Yang" <[e-mail]>]

Hi Lars,

Just posted an updated version, and a diff from the previous version is
available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-alto-performance-metrics-20

Thanks!
Richard

On Tue, Nov 30, 2021 at 8:49 PM Y. Richard Yang <[e-mail]> wrote:

> Hi Lars,
>
> Thanks for the review! Please see below.
>
> On Mon, Nov 29, 2021 at 8:10 AM Lars Eggert via Datatracker <
> [e-mail]> wrote:
>
>> Lars Eggert has entered the following ballot position for
>> draft-ietf-alto-performance-metrics-19: Discuss
>>
>> Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/
>> for more information about how to handle DISCUSS and COMMENT positions.
>>
>> The document, along with other ballot positions, can be found here:
>> https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/
>>
>>
>>
>> ----------------------------------------------------------------------
>> DISCUSS:
>> ----------------------------------------------------------------------
>>
>> This document needs to become much more formal about how it defines the
>> metrics it wishes to use with ALTO. This could either be done either by
>> identifying and normatively referencing existing metrics the IETF has
>> defined,
>> or by defining them here. When normatively referencing existing IETF
>> metrics, it
>> would need to explain why their use with ALTO makes sense.
>>
>> At the moment, the document informatively points to a somewhat arbitrary
>> collection of prior IETF metrics (most of which are from IPPM, residual
>> bandwidth from IS-IS TE, but then reservable bandwidth from OSPF TE?).
>
>
> To give some background, the WG derived the list of metrics from RFC 8571
> (BGP - Link State (BGP-LS) Advertisement of IGP Traffic Engineering
> Performance
> Metric Extensions), focusing on network->application. The list added Hop
> Count
> (exists in original ALTO RFC 7285), Round-trip (to avoid two queries, and
> many apps
> use RTT), and TCP Throughput, and removed Unidirectional Available
> Bandwidth
> and Unidirectional Utilized Bandwidth, to reduce the number of bandwidth
> metrics.
>
>
>> But it
>> only refers to them as "examples",
>
>
> I searched the word "example" and do not see where the document says that
> they
> are examples. It says that "Since different applications may use different
> cost metrics,
> the ALTO base protocol introduces an ALTO Cost Metric Registry (Section
> 14.2 of
> [RFC7285]), as a systematic mechanism to allow different metrics to be
> specified. For
> example, a delay-sensitive application may want to use latency-related
> metrics, and
> a bandwidth-sensitive application may want to use bandwidth-related
> metrics."
>
> Does this paragraph give an impression that the metrics are only examples?
> If so,
> do you suggest removing the "For example" phrase to reduce the impression?
>
> The document does have the sentence " The "Origin Example" column of Table
> 1 gives an example RFC that has defined each metric." Here the word
> "example"
> word means one existing work.
>
>
>> without actually defining how exactly they
>> are to be used with ALTO, or - if not those - which actual metrics are
>> supposed
>> to be used.
>>
>
> The document has "... the ALTO base protocol introduces an ALTO Cost
> Metric Registry
> (Section 14.2 of [RFC7285]), as a systematic mechanism to allow different
> metrics
>  to be specified. " and "When an ALTO server supports a cost metric
> defined in this document,
> it should announce this metric in its information resource directory (IRD)
> as defined in
>  Section 9.2 of [RFC7285]." Does this provide enough on how exactly they
> should be used?
> The function of this document is to satisfy the registry and the use will
> be in the base protocol
> (RFC7285). If there is a specific suggestion, it will be good to have.
>
>
>> Defining a mechanism for exposing metric information to clients isn't
>> really
>> useful unless the content of that information is much more clearly
>> specified.
>>
>> I agree with this statement that information should be specified as
> clearly as
> possible, but at the same time, we need abstraction to reduce the
> complexity.
> One guiding principle in the design is that ALTO information provides
> reasonable guidance, not mathematical precision.
>
>
>> Section 4.1.3. , paragraph 2, discuss:
>> >    Intended Semantics: To give the throughput of a TCP congestion-
>> >    control conforming flow from the specified source to the specified
>> >    destination; see [RFC3649, Section 5.1 of RFC8312] on how TCP
>> >    throughput is estimated.  The spatial aggregation level is specified
>> >    in the query context (e.g., PID to PID, or endpoint to endpoint).
>>
>> A TCP bandwidth estimate can only be meaningfully be derived for bulk TCP
>> transfers
>
>
> Yes. It is intended for bulk transfer.
>
>
>> under a set of pretty strict and simplistic assumptions, making this
>> metric a meaningless at best and misleading at worst,
>
>
> I will say that TCP throughput formula in general has turned out to be
> quite useful.
>
>
>
>> given that the source of
>> this information doesn't know what workload, congestion controller and
>> network
>> conditions the user of this information will use or see.
>>
>
> Network (the source) is in a pretty good position to estimate the
> potential TCP
> throughput. In a high multiplexing setting (small fish in a big pond),
> network can
> have access to estimated loss rate, RTT, and typical packet size to compute
> the TCP throughput formula. In a low multiplexing setting (big fish in a
> small pond),
> network can know the set of flows and estimate the bandwidth share. See
> the citation
> of the Prophet work in the document and the G2 work in SIGMETRICS'21 and
> SIGCOMM'21. The congestion controller info is part of the metric (the link
> points
> to standard TCP/Reno). I made some minor edits to clarify.
>
>
>> Also, RFC3649 is an Experimental RFC (from 2003!) and RFC8312 is an
>> Informational RFC. Since this document normatively refers to them, it
>> needs to
>> cite them, and this will cause DOWNREFs for PS document. I would argue
>> that
>> at least RFC3649 is certainly not an appropriate DOWNREF.
>>
>>
> Good suggestion! I added the reference to 3649 from the second paragraph of
> Sec. 5.1 of RFC8312 (you are a co-author). It reads "The average
> window sizes of Standard TCP and HSTCP are from [RFC3649].  The
> average window size of CUBIC is calculated using Eq. 6 and the CUBIC
> TCP-friendly region for three different values of C." Our plan, which is
> already suggested
> by Martin but it is my fault to not update yet, is to remove 3649 and use
> RFC8312bis.
> Make sense?
>
>
>> Why define this metric at all? The material you point to is the usual
>> model-based throughput calculation based on RTT and loss rates; a client
>> that
>> intended to predict TCP performance could simply query ALTO for this and
>> perform
>> their own computation, which will likely be more accurate, since the
>> client will
>> hopefully know which congestion controller they will use for the given
>> workload,
>> and what the characteristics of that workload are.
>>
>
> The throughput formula is for a very limited setting, i.e., the small fish
> setting. What
> we found useful is the low multiplexing setting, where the loss rate is
> the output,
> not the input, of the convergence process. It has good use cases. Please
> see
> the Prophet paper and one most recent example is the use cases, such as
> accelerating time-bound constrained flows, in Sec. 3 of
> https://www.reservoir.com/wp-content/uploads/2021/08/G2_QTBS_TR_2021.pdf
> The paper uses max-min fairness but the Internet uses other fairness.
>
>
>
>> ----------------------------------------------------------------------
>> COMMENT:
>> ----------------------------------------------------------------------
>>
>> Section 1. , paragraph 6, comment:
>> >    The purpose of this document is to ensure proper usage of the
>> >    performance metrics defined in Table 1; it does not claim novelty of
>> >    the metrics.  The "Origin Example" column of Table 1 gives an example
>> >    RFC that has defined each metric.
>>
>> I don't understand what the purpose of the "origin example" column is.
>> Most of
>> these point to IPPM metrics, which have a pretty clear and
>> narrowly-defined area
>> of applicability. Since ALTO isn't performing IPPM-style network testing,
>> it's
>> not clear why IPPM metrics are referenced here?
>>
>
> The metrics that this document use are defined in multiple IETF documents
> before.
> The intention of the sentence is to give early work credit.
>
>
>> Section 2.2. , paragraph 23, comment:
>> >    If a cost metric string does not have the optional statical operator
>> >    string, the statistical operator SHOULD be interpreted as the default
>> >    statical operator in the definition of the base metric.  If the
>>
>> What is a "statical" operator; I am not familiar with the term and it
>> doesn't
>> seem to appear in other RFCs? (Also occurs elsewhere in this document.)
>>
>> Apology for the typo. statical operator -> statistical operator. They are
> fixed in
> an internal version but we did not upload.
>
>
>
>
>> Section 3.1.4. , paragraph 4, comment:
>> >    link statistics.  Another example of a source to estimate the delay
>> >    is the IPPM framework [RFC2330].  It is RECOMMENDED that the
>>
>> IPPM defines measurement metrics. How would they be a source for
>> estimates?
>>
>>
> The intention was to refer to the measurement methodology in 6.2 of RFC
> 2330, but
> I can see the potential confusion now. How about we change the wording to
> "Another example of a source to estimate the delay is through active
> measurements,
> for example, considering the IETF IPPM framework [RFC2330]."
>
>
>> Section 3.3. , paragraph 1, comment:
>> > 3.3.  Cost Metric: Delay Variation (delay-variation)
>>
>> Is this supposed to apply to the one-way or bidirectional delay?
>
>
> This is the current specification: "
> 3.3.3.  Intended Semantics and Use
>
>    Intended Semantics: To specify spatial and temporal aggregated delay
>    variation (also called delay jitter)) with respect to the minimum
>    delay observed on the stream over the one-way delay from the
>    specified source and destination.  The spatial aggregation level is
>    specified in the query context (e.g., PID to PID, or endpoint to
>    endpoint)."
>
> So it is one way.
>
> Also, delay
>> variation is not independent from path utilization (c.f. bufferbloat), so
>> why is
>> it being reported independently?
>>
>
> Not sure I understand the suggestion. We see reports of jitter
> (e.g., https://cpr.att.com/pdf/se/0001-0003.pdf) reported independently
> (in the sense
> as a single metric, without specifying as conditional
> values/probabilities).
>
>
>>
>> Section 3.5. , paragraph 1, comment:
>> > 3.5.  Cost Metric: Loss Rate (lossrate)
>>
>> What is this metric supposed to capture? Loss is generally not
>> independent from
>> network utilization (apart from random corruption loss). So it should be
>> zero
>> for unloaded networks, and depends on utilization otherwise. Also, is this
>> unidirectional or bidirectional loss (wording below is unclear)?
>>
>
> It is meaningful in high multiplexing settings. There can also be an
> load-independent
> (I can see that you may see interference can be load as well) loss rate
> when there are
> wireless links.
>
> It is intended to be unidirectional: "3.5.3.  Intended Semantics and Use
>
>    Intended Semantics: To specify spatial and temporal aggregated packet
>    loss rate from the specified source and the specified destination.
>    The spatial aggregation level is specified in the query context
>    (e.g., PID to PID, or endpoint to endpoint)."
>
> How about the following change:
> " To specify spatial and temporal aggregated packet
>    loss rate from the specified source and the specified destination."
> =>
>  To specify spatial and temporal aggregated packet
>    loss rate, in one way, from the specified source and the specified
> destination."
>
>
> Using lowercase "not" together with an uppercase RFC2119 keyword is not
>> acceptable usage. Found: "MUST not"
>>
>>
> Got it. We have fixed the case:
> "The total length of the cost metric string MUST not exceed 32"
> =>
> "The total length of the cost metric string MUST NOT exceed 32"
>
>
>> The document has 6 authors, which exceeds the recommended author limit. I
>> assume the sponsoring AD has agreed that this is appropriate?
>>
>> No reference entries found for: [RFC3649] and [RFC8312].
>>
>>
> Thanks for pointing it out. It was missing after an update and pointed out
> by
> Martin. It is fixed in the next version which we will upload soon.
>
>
>> Found terminology that should be reviewed for inclusivity; see
>> https://www.rfc-editor.org/part2/#inclusive_language for background and
>> more
>> guidance:
>>
>>  * Term "man"; alternatives might be "individual", "people", "person".
>>
>> Hah. You mean change
> "man-in-the-middle (MITM) attacks"
> =>
> "person-in-the-middle attacks".
>
> I looked and indeed see PITM (
> https://en.wikipedia.org/wiki/Man-in-the-middle_attack).
>
> Interesting and fixed. Thanks!
>
> The edits below are great and fixed. Thanks again!
>
> Richard
>
>
>
>> -------------------------------------------------------------------------------
>> All comments below are about very minor potential issues that you may
>> choose to
>> address in some way - or ignore - as you see fit. Some were flagged by
>> automated tools (via https://github.com/larseggert/ietf-reviewtool), so
>> there
>> will likely be some false positives. There is no need to let me know what
>> you
>> did with these suggestions.
>>
>> "Abstract", paragraph 2, nit:
>> -    types of cost metric.  Since the ALTO base protocol (RFC 7285)
>> +    types of cost metrics.  Since the ALTO base protocol (RFC 7285)
>> +                        +
>>
>> Section 1. , paragraph 2, nit:
>> > ] on registering ALTO cost metrics. Hence it specifies the identifier,
>> the in
>> >                                     ^^^^^
>> A comma may be missing after the conjunctive/linking adverb "Hence".
>>
>> Section 2.2. , paragraph 2, nit:
>> > of the observations. median: the mid point (i.e., p50) of the
>> observations.
>> >                                  ^^^^^^^^^
>> This word is normally spelled with a hyphen.
>>
>> "IPPM ", paragraph 2, nit:
>> >  Also, delay variation is not independent from path utilization (c.f.
>> buffer
>> >                               ^^^^^^^^^^^^^^^^
>> The usual collocation for "independent" is "of", not "from". Did you mean
>> "independent of"?
>>
>> Section 3.3.3. , paragraph 7, nit:
>> > apture? Loss is generally not independent from network utilization
>> (apart fr
>> >                               ^^^^^^^^^^^^^^^^
>> The usual collocation for "independent" is "of", not "from". Did you mean
>> "independent of"?
>>
>> Section 3.4.3. , paragraph 6, nit:
>> > imation" method. See Section 3.1.4 on on related discussions such as
>> summing
>> >                                    ^^^^^
>> Possible typo: you repeated a word.
>>
>> Section 3.5.4. , paragraph 3, nit:
>> >  [RFC8312]), it helps to specify as much details as possible on the the
>> cong
>> >                                     ^^^^
>> Use "many" with countable plural nouns like "details".
>>
>> Section 3.5.4. , paragraph 3, nit:
>> > ify as much details as possible on the the congestion control algorithm
>> used
>> >                                    ^^^^^^^
>> Two determiners in a row. Choose either "the" or "the".
>>
>> These URLs in the document can probably be converted to HTTPS:
>>  *
>> http://www.iana.org/assignments/alto-protocol/alto-protocol.xhtml#cost-metrics
>>
>>
>>
>> _______________________________________________
>> alto mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/alto
>
>