Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

["MORTON JR., AL" <[e-mail]>]

Hi Christoph,

you wrote:
> I think what would be helpful would be a section in the draft that explains the different sources of latency (network, server, client) and how they affect the final RPM-number and how one can separate out these two components. It is also important to understand that the results are highly implementation-dependent. And explaining that in this section should help, I believe.
>

when you say, "...the results are highly implementation-dependent.", 
which implementations are you referring to:
 
Network path? Server? Client? "Working Load" conditions? 

All of the above?

Network implementation should influence the results, the others not so much (with a standardized metric, we should have equivalent results across implementations, at least that's what we said in RFC6576/BCP176). And it can be rather difficult to separate the (I count at least 4) results-influencing factors.

more stuff to consider,
Al


> -----Original Message-----
> From: ippm <[e-mail]> On Behalf Of Dave Taht
> Sent: Tuesday, February 15, 2022 8:00 PM
> To: Christoph Paasch <cpaasch=[e-mail]>
> Cc: Marcus Ihlar <marcus.ihlar=[e-mail]>; IETF IPPM WG
> <[e-mail]>
> Subject: Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness
> 
> On Tue, Feb 15, 2022 at 4:11 PM Christoph Paasch
> <cpaasch=[e-mail]> wrote:
> >
> > Hello Greg,
> >
> > On Feb 8, 2022, at 1:10 PM, Greg Mirsky <[e-mail]> wrote:
> >
> > Hi Christoph,
> > apologies for the belated response and thank you for sharing interesting
> details of you using the measurement method. I think that if the measurement
> method can not only provide the Round-trip Per Minute (RPM) metric but expose
> the network propagation and residential components of the round-trip delay,
> then it seems to me, the scope of the draft to be aligned with the charter of
> the IPPM WG and I'll be in favor of the WG adoption of the work.
> > What do you think? What is the opinion of the authors and the WG?
> >
> >
> > I am assuming that with "residential components" you mean the server/client-
> side contribution to the measured latency, right?
> >
> > In that case, yes the method does allow to separate these, as latency-probes
> are sent on both the load-generating connections and on separate connections.
> The difference between the two represents the "server-side contribution" to
> the latency.
> 
> I remember vividly how "neat" it seemed that QUIC had adopted the idea
> of an inband "ping" in version Q018 in 2012 (or so).
> 
> https://urldefense.com/v3/__https://docs.google.com/document/d/1WJvyZflAO2pq77
> yOLbp9NsGjC1CHetAXV8I0fQe-B_U/edit__;!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPeUiZuj5$
> Q018: Added a PING frame
> 
> What I don't remember or understand at the moment is I'm under the
> impression that's in the SSL layer as of http2.0? There a good ref?
> 
> 
> 
> >
> > I think what would be helpful would be a section in the draft that explains
> the different sources of latency (network, server, client) and how they affect
> the final RPM-number and how one can separate out these two components. It is
> also important to understand that the results are highly implementation-
> dependent. And explaining that in this section should help, I believe.
> >
> > Would that be in line with what you are looking for?
> >
> >
> > Thanks,
> > Christoph
> >
> >
> >
> > Regards,
> > Greg
> >
> > On Thu, Jan 6, 2022 at 4:42 PM Christoph Paasch <[e-mail]> wrote:
> >>
> >> Hello Greg,
> >>
> >> On Jan 6, 2022, at 3:00 PM, Greg Mirsky <[e-mail]> wrote:
> >>
> >> Hi Christoph,
> >> a happy and healthy New Year to you and All!
> >>
> >>
> >> Happy New Year to you as well!
> >>
> >> Thank you for your kind consideration of my notes and detailed responses.
> Please find my follow-up notes in-line below under the GIM>> tag.
> >>
> >>
> >> Thanks for your replies. Please see inline:
> >>
> >> On Wed, Jan 5, 2022 at 10:52 AM Christoph Paasch <[e-mail]> wrote:
> >>>
> >>> Hello Greg,
> >>>
> >>> thanks for your comments. Please see inline:
> >>>
> >>> On Dec 22, 2021, at 11:43 PM, Greg Mirsky <[e-mail]> wrote:
> >>>
> >>> Dear Marcus, Authors, et al,
> >>> apologies for the belated response.
> >>> I've read the draft and have some comments to share with you:
> >>>
> >>> as I understand it, the proposed new responsiveness metric is viewed as
> the single indicator of a bufferbloat condition in a network. As I recall, the
> discussion at Measuring Network Quality for End-Users workshop and on the
> mailing list indicated, that there’s no consensus on what behaviors, symptoms
> can reliably signal the bufferbloat.
> >>>
> >>> We are not trying for this responsiveness metric to be "the single
> indicator of bufferbloat". Bufferbloat can be measured in many different
> number of ways. And each of these will produce a correct, but a different
> result. Thus, "bufferbloat" is whatever the methodology tries to detect.
> >>>
> >>> Let me give an example of two methodologies that are both correct but both
> will produce entirely different numbers :
> >>>
> >>> If we would decide to generate the load by flooding the network with UDP
> traffic from a specific 4-tuple and measure latency with parallel ICMP pings.
> Then, on a over-buffered FIFO queue we would measure huge latencies (thus
> correctly expose bufferbloat), while on a FQ-codel queue we would not measure
> any bufferbloat.
> >>>
> >>> If on the other hand, the load-generating traffic is changing the source-
> port for every single UDP-packet, then in both the FIFO-queue and the FQ-codel
> queue we will measure huge amounts of bufferbloat.
> >>>
> >>> Thus, these two methods both produced correct results but with hugely
> different numbers in the FQ-codel case. [1]
> >>>
> >>> Now, while both methods measure some variant of bufferbloat, they both
> don't measure a realistic usage of the network.
> >>
> >> GIM>> Thank you for the insights. It seems to me that what the method can
> demonstrate is rather the level of efficiency of the AQM in the network for a
> particular class of applications.
> >>
> >>
> >> Yes, that is a good description. It is for a "particular class of
> applications" and we are trying to make this class of applications
> representative of a "typical user-scenario". (admittedly, we can debate
> forever on what kind of applications are representative and I would love to
> have that debate :-)).
> >>
> >> On the point of "efficiency of the AQM". I would go even further that it's
> not only AQM but also the client- and server-side implementations of these
> applications (as noted further below).
> >>
> >>
> >>>
> >>>
> >>> That is why the "Responsiveness under working conditions" tries to clearly
> specify how the load is generated and how the latency is being measured. And
> it does not measure "bufferbloat" but it measures "responsiveness under
> working conditions" based on the methodology that is being used (using HTTP/2
> or HTTP/3, multiple flows, ...). It does expose bufferbloat which can happen
> in the network. It also exposes certain server-side behaviors that can cause
> (huge amounts of) additional latency - those behaviors are typically not
> called "bufferbloat".
> >>
> >> GIM>> Thank you for pointing out that the result of the RTT measurement has
> two contributing factors - network and server.
> >>
> >>
> >> Yes, servers contribute as do the client-side implementations. It's all
> three (client, network, server) that need to work "correctly" to achieve good
> responsiveness. Btw., as we are now gathering more experience with our
> methodology in different environments we find that the biggest portions of
> latency actually come from the server-side. We see several seconds of latency
> introduced by the HTTP/2 and TCP implementations.
> >>
> >> It seems worth enhancing the method to localize each contribution and
> measure them separately.
> >>
> >>
> >> With the latency measuring probes being sent on load-bearing connections
> and separate connections and with the separate connections serving to measure
> DNS/TCP/... individually, the different data-points actually allow to localize
> to some extend.
> >>
> >> However, I would be reluctant to dive too deep into localization/trouble-
> shooting/debugging of networks as part of this I-D. As this opens a whole new
> can of worms. We could then start thinking about sending latency-probes while
> playing with the IP TTL to find which router is introducing the latency,...
> It's an entirely different research-topic IMO :-) Dave Taht was thinking of
> starting something along these lines
> (https://urldefense.com/v3/__https://github.com/dtaht/wtbb__;!!BhdT!yrleDfru-
> 45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPRghwEjg$ ).
> >>
> >>> It seems that it would be reasonable to first define what is being
> measured, characterized by the responsiveness metric. Having a document that
> discusses and defines the bufferbloat would be great.
> >>>
> >>> I agree that there is a lack of definition for what "bufferbloat" really
> is.
> >>>
> >>> The way we look at "responsiveness under working conditions" is that it
> measures the latency in conditions that may realistically happen in worst-case
> scenarios with end-users/implementations that are non-malicious (non-malicious
> to exclude the UDP-flooding scenario).
> >>>
> >>> Thus, I assume we should make a better job at explaining this. The lack of
> a formal definition of "bufferbloat" doesn't help and thus we are indeed using
> this term a bit freely in the current draft. We will improve the Introduction
> to better set the stage
> (https://urldefense.com/v3/__https://github.com/network-quality/draft-cpaasch-
> ippm-responsiveness/issues/31__;!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPU0pYBOv$ ).
> >>>
> >>> It seems like in the foundation of the methodology described in the draft
> lies the assumption that without adding new flows the available bandwidth is
> constant, does not change. While that is mostly the case, there are
> technologies that behave differently and may change bandwidth because of the
> outside conditions. Some of these behaviors of links with variable discrete
> bandwidth are discussed in, for example, RFC 8330 and RFC 8625.
> >>>
> >>> I'm not sure I entirely understand your comment. But let me explain why we
> are gradually adding new flows:
> >>>
> >>> 1. TCP-implementations have usually a fixed limit for the upper bound of
> the receive window. In some networks that upper bound is lower than the BDP of
> the network. Thus, the only way to reach full capacity is by having multiple
> flows.
> >>> 2. Having multiple connections allows to quicker achieve full capacity in
> high-RTT networks and thus speeds up the test-duration.
> >>> 3. In some networks with "random" packet-loss, congestion-control may come
> in the way of achieving full capacity. Again, multiple flows will work around
> that.
> >>
> >> GIM>> I might have asked several questions at once. Let me clarify what I
> am looking for:
> >>
> >> As I understand the method of creating the "working conditions in a
> network" is based on certain assumptions. First, seems is that the bandwidth
> is symmetrical between the measurement points. Second, that the bandwidth
> doesn't change for the duration of the measurement session. AFAIK, in the
> access networks, both are not necessarily always the case.
> >>
> >> We don't have the assumption that bandwidth is symmetrical (assuming, you
> mean uplink/downlink symmetry - please clarify otherwise).
> >>
> >> The load-generating algorithm runs independently for uplink and downlink
> traffic. And it is perfectly fine when both have huge asymmetry.
> >>
> >>
> >> Regarding the stability of the bandwidth:
> >> You are making a good point indeed that we assume that the bandwidth is to
> some extend stable while ramping up the flows to "working conditions".
> Admittedly that assumption does not always hold, and that is one of the
> reasons why we try hard for the test to not take too long.
> >> I'm not sure how we could adjust the algorithm for varying bandwidth
> without introducing too much complexity. I'm open for suggestions :-)
> >>
> >> On the other hand, I might have missed how the method of creating the
> "working conditions" guarantees a symmetrical load between the measurement
> points.
> >>
> >> As mentioned above, we don't assume a symmetrical load. Can you show us
> where in the draft we give that impression, so we can fix that?
> >>
> >>>
> >>> Then, I find the motivation not to use time units to express the
> responsiveness metric not convincing:
> >>>
> >>>    "Latency" is a poor measure of responsiveness, since it can be hard
> >>>    for the general public to understand.  The units are unfamiliar
> >>>    ("what is a millisecond?") and counterintuitive ("100 msec - that
> >>>    sounds good - it's only a tenth of a second!").
> >>>
> >>>
> >>> Can you expand on what exactly is not convincing to you? Do you think that
> people will mis-understand the metric or that milli-seconds is the right way
> to communicate responsiveness to the general public?
> >>
> >> GIM>> Let me try. We know packet delay requirements for AR, VR
> applications. I believe that gamers are familiar with these numbers too. The
> same is likely the case for the industrial automation use cases served, for
> example, by Deterministic Networking.
> >>
> >>
> >> I can understand that for a technical audience, milli-seconds is easy and
> familiar. A non-technical audience might be more open to accepting a new
> "higher-is-better" metric. Responsiveness is something new and abstract so,
> it's kind of natural that it comes with a new unit.
> >>
> >> But I fully recognize that that's a controversial topic and can be
> discussed at length :)
> >>
> >>
> >> Cheers,
> >> Christoph
> >>
> >>>
> >>>
> >>> Thanks a lot,
> >>> Christoph
> >>>
> >>> [1] And there are many networks that prioritize ICMP pings, thus we could
> observe even more different results based on what protocol is used to measure
> the latency.
> >>>
> >>>
> >>> On Mon, Dec 6, 2021 at 7:53 AM Marcus Ihlar
> <marcus.ihlar=[e-mail]> wrote:
> >>>>
> >>>> Hi IPPM,
> >>>>
> >>>>
> >>>>
> >>>> This email starts an adoption call for draft-cpaasch-ippm-responsiveness,
> "Responsiveness under Working Conditions”. This document specifies the “RPM
> Test” for measuring user experience when the network is fully loaded. The
> intended status of the document is Experimental.
> >>>>
> >>>>
> >>>>
> >>>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-
> cpaasch-ippm-responsiveness/__;!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPU3zpKI3$
> >>>>
> >>>> https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-
> cpaasch-ippm-responsiveness-01__;!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPf9N-E8h$
> >>>>
> >>>>
> >>>>
> >>>> This adoption call will last until Monday, December 20. Please review the
> document, and reply to this email thread to indicate if you think IPPM should
> adopt this document.
> >>>>
> >>>>
> >>>>
> >>>> BR,
> >>>>
> >>>> Marcus
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ippm mailing list
> >>>> [e-mail]
> >>>>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$
> >>>
> >>> _______________________________________________
> >>> ippm mailing list
> >>> [e-mail]
> >>>
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$
> >>>
> >>>
> >>
> >
> > _______________________________________________
> > ippm mailing list
> > [e-mail]
> >
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$
> 
> 
> 
> --
> I tried to build a better future, a few times:
> https://urldefense.com/v3/__https://wayforward.archive.org/?site=https*3A*2F*2
> Fwww.icei.org__;JSUl!!BhdT!yrleDfru-45Lar4Qe5-
> 4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPXLNE64R$
> 
> Dave Täht CEO, TekLibre, LLC
> 
> _______________________________________________
> ippm mailing list
> [e-mail]
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ippm__;!!Bhd
> T!yrleDfru-45Lar4Qe5-4bWSRTYtJSdvp8oh643W10p69kHM1mzsvPTFWzNHi$