Re: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness

[Christoph Paasch <[e-mail]>]

And I just realize I jumped over some parts of your comments. Please see more inline:

> On Jan 4, 2022, at 4:23 PM, Christoph Paasch <cpaasch=[e-mail]> wrote:

[snip]

>>  
>>  
>> Finally, in 4.1.4, the Note explains:
>>  
>>    Note: It is tempting to envision an initial base RTT measurement and
>>    adjust the intervals as a function of that RTT.  However, experiments
>>    have shown that this makes the saturation detection extremely
>>    unstable in low RTT environments.  In the situation where the
>>    "unloaded" RTT is in the single-digit millisecond range, yet the
>>    network's RTT increases under load to more than a hundred
>>    milliseconds, the intervals become much too low to accurately drive
>>    the algorithm.
>>  
>> Well, TCP senders/control-loops are involved here, and likely play a
>> role in behavior categorized as “difficult to measure”.
>>  
>> By the time we get to
>>  
>> 4.2 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01#section-4.2>.  Measuring Responsiveness
>>  
>>    Once the network is in a consistent working conditions, the RPM Test
>>    must "probe" the network multiple times to measure its
>>    responsiveness.
>>  
>>    Each RPM Test probe measures:
>>  
>> You previously started at least four TCP connections with infinitely large files.
>> The “create connection” RPM probes establish additional connections, DNS, TCP, etc.
>> Is each new connection an RPM probe? or is the set of connection tests a single probe?
>> (later we learn it is the set of
>> What if one of the set of connections fails/times-out?

Yes, the set will be factored into the final RPM number.

>>  
>> I take it that the “load-bearing” connections are driving the path to saturation.
>> Maybe “load-generating connections” is more clear?
> 
> "load-generating" is indeed a better term!
> 
> For all of the above "minor" edits, I filed https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/35 <https://github.com/network-quality/draft-cpaasch-ippm-responsiveness/issues/35> 
> 
>> 4.2.1 <https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01#section-4.2.1>.  Aggregating the Measurements
>>  
>>    The algorithm produces sets of 5 times for each probe, namely: DNS
>>    handshake, TCP handshake, TLS handshake, HTTP/2 request/response on
>>    separate (idle) connections, HTTP/2 request/response on load bearing
>>    connections.  This fine-grained data is useful, but not necessary for
>>    creating a useful metric.
>>  
>> @@@@ So, only ONE of the load-generating connections runs the 1-byte GET ?  (it says connections, and there are at least 4) The various handshakes result in 4 RT measurements.
>>  
>> But, do you mean *5 repeated measurement sets*? Each set with:
>> DNS HS, TCP HS, TLS HS, HTTP/2 idle GET, and the potentially much longer GET on the 
>> load generating connections.

You are right, this is not very clear. I am working on testing a refinement of the methodology and how the numbers are aggregated.

I'm thinking of the following:

Each of the RPM-measuring separate connections should get latency numbers for DNS, TCP, TLS and H2 req/response. Additionally, we have the measurement on the load-generating connections (one probe on each of these).

This gives us 5 sets of numbers:
DNS: [42ms, 39ms, 43ms, ...] -> 75th percentile = V
TCP: [..., ..., ...] -> 75th percentile = W
TLS: [... ,..., ...] -> ... = X
H2: [..., ..., ...] ->  ... = Y
H2 on load-generating: [..., ..., ...] -> ... = Z

Then, RPM is the average (normalized to 60 seconds) of these values.

The tricky part is to choose 75th percentile or 90th percentile,... as well as how many probes are needed.


Does this make it clearer on how the probes are measured and create the RPM-value ?


Christoph

>>    To create a single "Responsiveness" (e.g., RPM) number, this first
>>    iteration of the algorithm gives an equal weight to each of these
>>    values.  That is, it sums the five time values for each probe, and
>>    divides by the total number of probes to compute an average probe
>>    duration.  The reciprocal of this, normalized to 60 seconds, gives
>>    the Round-trips Per Minute (RPM).
>>  
>> @@@@ I’m missing a step, I think:
>> Are the “time values for each probe” the sum of handshake or response times for
>> DNS HS, TCP HS, TLS HS, HTTP/2 idle GET and load generating connections?
>> The processing doesn’t seem to include this preliminary calculation to produce 
>> “five time values”.  
>>  
>>  
>> In Section 5, “no new protocol is defined”, but
>>  
>>    The client begins the responsiveness measurement by querying for the
>>    JSON configuration.  This supplies the URLs for creating the load
>>    bearing connections in the upstream and downstream direction as well
>>    as the small object for the latency measurements.
>>  
>> The client needs to know how the server response will be organized, down to key: value, right?
>> Some client and server agreements needed...
> 
> Yes! With "protocol" we mean nothing new at HTTP/TCP/... layer. Just standard JSON. I don't know if that qualifies as a new "protocol" ;-)
> 
> 
> Thanks a lot,
> Christoph
> 
> 
>>  
>>  
>> From: ippm <[e-mail] <mailto:[e-mail]>> On Behalf Of Marcus Ihlar
>> Sent: Monday, December 6, 2021 10:53 AM
>> To: [e-mail] <mailto:[e-mail]>
>> Subject: [ippm] Adoption call for draft-cpaasch-ippm-responsiveness
>>  
>> Hi IPPM,
>>  
>> This email starts an adoption call for draft-cpaasch-ippm-responsiveness, "Responsiveness under Working Conditions”. This document specifies the “RPM Test” for measuring user experience when the network is fully loaded. The intended status of the document is Experimental.   
>>  
>> https://datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/ <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-cpaasch-ippm-responsiveness/__;!!BhdT!zI6d5je1i8cafA6NXByD5tvxHFKKPMjYgtM6t2aLUHFPsyPz-XwPFguwa1HS$>
>> https://datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01 <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-cpaasch-ippm-responsiveness-01__;!!BhdT!zI6d5je1i8cafA6NXByD5tvxHFKKPMjYgtM6t2aLUHFPsyPz-XwPFq67PfWg$>
>>  
>> This adoption call will last until Monday, December 20. Please review the document, and reply to this email thread to indicate if you think IPPM should adopt this document.
>>  
>> BR,
>> Marcus 
>>  
>> _______________________________________________
>> ippm mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/ippm <https://www.ietf.org/mailman/listinfo/ippm>
> _______________________________________________
> ippm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ippm