Re: [ippm] review: draft-ietf-ippm-responsiveness-03

[Christoph Paasch <[e-mail]>]

Hello Sebastian,

> On Jan 18, 2024, at 12:48 AM, Sebastian Moeller <moeller0=[e-mail]> wrote:
>> On 18. Jan 2024, at 00:46, Christoph Paasch <cpaasch=[e-mail]> wrote:
>>> On Jan 17, 2024, at 12:03 PM, Sebastian Moeller <moeller0=[e-mail]> wrote:
>>>> 
>>>> On 17. Jan 2024, at 18:55, Christoph Paasch <cpaasch=[e-mail]> wrote:
>>>>> On Dec 10, 2023, at 7:50 AM, Sebastian Moeller <moeller0=[e-mail]> wrote:
>>>>> 4.3.1.  Aggregating the Measurements
>>>>> 
>>>>> The algorithm produces sets of 4 times for each probe, namely: tcp_f,
>>>>> tls_f, http_f, http_l (from the previous section).  The
>>>>> responsiveness of the network connection being tested evolves over
>>>>> time as buffers gradually reach saturation.  Once the buffers are
>>>>> saturated, responsiveness will stabilize.  Thus, the final
>>>>> calculation of network responsiveness considers the last MAD (Moving
>>>>> Average Distance - default to 4) intervals worth of completed
>>>>> responsiveness probes.
>>>>> 
>>>>> NOTE: for a network limited by MPS that means 400 latency samples, which is pretty impressive! For a PTC limited network however that could be considerably less. Could we require a verbose mode that reports the number of latency samples taken into account and that also reportes some measure of variance?
>>>> 
>>>> Implementations are free to report additional information. I’m not sure whether that should be specified in an IETF RFC, because the RFC defines the methodology only, no?
>>> 
>>> [SM] Well, there is this idea that if one reports aggregate data one should also give an idea how well that data holds, and often that starts with the number of data points included, something more and more on-line tests start to reveal (e.g. by allowing to download a csv of the detailed results, which includes the individual samples, so getting to the N just requires counting :) )
>> 
>> Sure. But this is IMO a tool/implementation-question.
> 
> [SM] Not really, IMHO this as validity of reporting question ;) The way things are phrased ATM, I believe the lowest possible number of samples for self might be MAD * 2, so 8 samples (we need at least 2 samples to calculate a standard deviation), while the expected number for most users will be closer to ~400, that is a really wide range and hence giving some indication seems really desirable. In the same line of thought, we might consider not only reporting the mean RPM but also a measure of deviation. To keep the default recommended result display clean and simple, I think this could be moved to a verbose ode if a client offers that, but if there is no verbose mode this should be reported (potentially as part of the confidence report). But I might be biassed here from requirements at work, and do not want to claim I am offering a universal truth here, but I do want to be sure what ever the draft recommends in this direction is a conscious decision and not an oversight. So what ever you decide here, I am fine with.

After some more offline discussions I am going to add a “additional metrics” subsection for verbose reporting in the “Interpreting responsiveness results” section.

> 
> 
>> 
>>> 
>>> [SM] True, as long as the draft does not imply that only 20 seconds is permitted, I am happy.
>> 
>> “20 seconds is permitted” ? I guess you mean “20 seconds is mandated”. Yes, we are not mandating 20 seconds. In general, everything is “permitted” :)
> 
> [SM] Yes, that is what I meant with "only 20 seconds is permitted"... I have a feeling quite a number of RFC based implementations stick really close to the text and do not think outside of the box... (case ion point, there is zero reason, why ICMPv6 should not have supported the base types that ICMPv4 supports like timestamps requests, but that was not stated explicitly in the relevant RFC as far as I can see, and now ICMPv6 lost the ability to use timestamps).

I get your point. Let me make sure the 20 seconds does not stand out as “this is the limit that you should implement”. (will be all folded into the PR at https://github.com/network-quality/draft-ietf-ippm-responsiveness/pull/97).

> 
> 
>> 
>>>>> We define "High" confidence if the algorithm was able to fully reach
>>>>> stability based on the defined standard deviation tolerance.
>>>>> 
>>>>> It must be noted that depending on the chosen standard deviation
>>>>> tolerance or other parameters of the methodology and the network-
>>>>> environment it may be that a measurement never converges to a stable
>>>>> point.  This is expected and part of the dynamic nature of networking
>>>>> and the accompanying measurement inaccuracies.  Which is why the
>>>>> importance of imposing a time-limit is so crucial, together with an
>>>>> accurate depiction of the "confidence" the methodology was able to
>>>>> generate.
>>>>> 
>>>>> QUESTION: Should we propose a recommended verbiage to convey the confidence to the user, to make different implementations easier to compare?
>>>> 
>>>> I think we did that, no? “Low” “Medium” and “High”.
>>> 
>>> [SM] I was daft and was expecting something like "The confidence score should be reported to the user as part of the main results", but I guess that is the implicit message of this section?
>> 
>> In my opinion it is fairly clear in the draft. Feel free to send a PR if you have an idea on how to make it more clear.
> 
> [SM] Fine then, I just note that goresponsiveness currently does not report that (I did not check it against all other parts of the draft yet, and there is still lots of progress in the go code, so it might have leaned that already ;) )

Sorry - now I understand what you meant. That the draft should say that an implementation should implement the confidence and report it to the user.

I am adding this!

>> 
>>>>> 6.  Responsiveness Test Server API
>>>>> 
>>>>> The responsiveness measurement is built upon a foundation of standard
>>>>> protocols: IP, TCP, TLS, HTTP/2.  On top of this foundation, a
>>>>> minimal amount of new "protocol" is defined, merely specifying the
>>>>> URLs that used for GET and PUT in the process of executing the test.
>>>>> 
>>>>> Both the client and the server MUST support HTTP/2 over TLS.  The
>>>>> client MUST be able to send a GET request and a POST.  The server
>>>>> MUST be able to respond to both of these HTTP commands.  The server
>>>>> MUST have the ability to provide content upon a GET request.  The
>>>>> server MUST use a packet scheduling algorithm that minimizes internal
>>>>> queueing to avoid affecting the client's measurement.
>>>>> 
>>>>> QUESTION: While I fully agree that is what a server should do, I would prefer to get numbers over not getting numbers... so maybe make this a SHOULD, also if the server needsto return time the request was received and time the response was sent out, maybe we can factor out the local processing?
>>>> 
>>>> Agree’d on the SHOULD use a packet scheduling […].
>>>> 
>>>> Regarding the time - see my response in the other thread.
>>>> 
>>>>> 
>>>>> 
>>>>> As clients and servers become deployed that use L4S congestion
>>>>> control (e.g., TCP Prague with ECT(1) packet marking), for their
>>>>> normal traffic when it is available, and fall back to traditional
>>>>> loss-based congestion controls (e.g., Reno or CUBIC) otherwise, the
>>>>> same strategy SHOULD be used for Responsiveness Test traffic.  This
>>>>> is RECOMMENDED so that the synthetic traffic generated by the
>>>>> Responsiveness Test mimics real-world traffic for that server.
>>>>> 
>>>>> NOTE: This clearly needs to be under end-user control, that is there should be a way for the user to request/enforce either option (I am fine with the default being, what the local OS defaults to). L4S is an experimental RFC and is is expected to have odd failure modes so a L4S aware measurement tool should keep in mind that L4S itself needs to be measured/tested. At the very least the results need to report whether L4S was used or not.
>>>> 
>>>> IMO, this is an implementation choice, whether a tool exposes configuration options to enable/disable L4S and what kind of additional statistics it exposes.
>>> 
>>> [SM] Mmmh, I am a bit uneasy with that, as I said it should report if a test was using L4S or not, I accept that maybe RPM is not the best tool for A/B testing L4S ;)
>> 
>> In a verbose mode there are a lot of things that can be exposed. L4S on/off. IPv4/IPv6. Once we go down this “rabbit hole” of what else could be exposed, we will start listing every single bit along the protocol stack that may theoretically have an impact on the responsiveness.
> 
> [SM] Clearly not a black and white matter given the multitude of potentially relevant parameters, however the client will know whether L4S was used or not, while it is quite hard for a user to figure this out (I think one would need to get packet captures and then look into the headers and connection negotiation).

Yes - see above. Will add some non-exhaustive suggestions on what additional passive metrics to report.


Thanks,
Christoph