Re: [ippm] Preliminary measurement comparison of "Working Latency" metrics

["MORTON JR., AL" <[e-mail]>]

Hi Randall, thanks for chiming-in!
There are additional test results in follow=up messages beyond the first set I shared below, where I tried networkQuality’s command-line options, and even goresponsiveness.

please see brief replies below [acm],
Al

From: Randall Meyer <[e-mail]>
Sent: Tuesday, November 15, 2022 5:53 PM
To: MORTON JR., AL <[e-mail]>
Cc: [e-mail]
Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency" metrics

Hello!

   Speaking for networkQuality, I might be able to add some color here.  The capacity values output from the tool are actually goodput and not raw throughput that I think Ookla and UDPST are showing.
[acm]
Since TCP handles the byte-stream delivery function in both cases, retransmitted and duplicate packets are discarded and it seems that networkQuality and Ookla would be measuring similar TCP=payload streams.
UDPST reports IP-Layer Capacity, and we measure packet duplication if it occurs (none in these measurements), so I think the values are comparable with the correction needed between IP-layer (we count the header bits) and TCP payload.

If you’re wanting to compare download-only goodput or RPM values, I would also suggest running networkQuality with -s (along with -v) to get a closer apples to apples comparison to Ookla or your own udpst. “networkQuality -s” will run upload and download tests sequentially.
[acm]
Thanks, that’s exactly what I did, networkQuality -vs , in the most recent testing.

In a subsequent message, you mentioned having to use Apple’s servers to use networkQuality or Will’s goresponsiveness. If you want to test on your own infrastructure,  we do provide some sample code on github. I recommend the go-server for ease of deployment/hacking on (https://github.com/network-quality/server/tree/main/go<https://urldefense.com/v3/__https:/github.com/network-quality/server/tree/main/go__;!!BhdT!gDy8HkQIlHamGvp43OZf-RS56h-swktmWW8SG6_ByXuWF4-z511HIzo4YYOTNFe6yUht7A$>) The default default networkQuality servers we run should be reasonably located to you in Chicago.
[acm]
ok, I’ll keep that possibility in mind.

Our UDPST server is back in NJ, and I measure ~28ms minimum RTT to our server. The minimum delay to Apple’s servers seems to be ~8ms, which is low. Ookla’s typical ping delay is about the same. I don’t think I can do better than Apple’s servers when testing networkQuality, especially with TCP’s RTT sensitivity.

As for the low RPM scores, it very well could be hardware. I’ve been comparing a couple of DOCSIS 3.1 modems with 1.2 Gbps provisioned, and one definitely performs worse than the other when doing simultaneous upload/download tests.
[acm]
Yes, the simultaneous test is stressful!

When you look at the more recent test results (and I have another set that I haven’t typed-up yet) using -vs, you’ll see that networkQuality measured approx the same downlink capacity (~890) in the simultaneous case below, and the sequential cases later. Seems like the DOCSIS HW I’m testing may be performing reasonably.

thanks again for joining the discussion!

-rm



On Oct 24, 2022, at 3:35 PM, MORTON JR., AL <[e-mail]<mailto:[e-mail]>> wrote:

Hi RPM friends and IPPM'ers,

I was wondering what a comparison of some of the "working latency" metrics would look like, so I ran some tests using a service on DOCSIS 3.1, with the downlink provisioned for 1Gbps.

I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla Speedtest with as similar connectivity as possible (but we know that the traffic will diverge to different servers and we can't change that aspect).

Here's a quick summary of yesterday's results:

Working Latency & Capacity Summary

Net Qual                UDPST                        Ookla
DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap    Ping(no load)
878       62            970      28       0-19       941      6
891       92            970      27       0-20       940      7
891       120           966      28       0-22       937      9
890       112           970      28       0-21       940      8
903       70            970      28       0-16       935      9

Note: all RPM values were categorized as Low.

networkQuality downstream capacities are always on the low side compared to others. We would expect about 940Mbps for TCP, and that's mostly what Ookla achieved. I think that a longer test duration might be needed to achieve the actual 1Gbps capacity with networkQuality; intermediate values observed were certainly headed in the right direction. (I recently upgraded to Monterey 12.6 on my MacBook, so should have the latest version.)

Also, as Sebastian Moeller's message to the list reminded me, I should have run the tests with the -v option to help with comparisons. I'll repeat this test when I can make time.

The UDPST measurements of RTTmin (minimum RTT observed during the test) and the range of variation above the minimum (RTTVarRnge) add-up to very reasonable responsiveness IMO, so I'm not clear why RPM graded this access and path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago conducting tests, so the minimum 28ms is typical. UDPST measurements were run on an Ubuntu VM in my MacBook.

The big disappointment was that the Ookla desktop app I updated over the weekend did not include the new responsiveness metric! I included the ping results anyway, and it was clearly using a server in the nearby area.

So, I have some more work to do, but I hope this is interesting-enough to start some comparison discussions, and bring-out some suggestions.

happy testing all,
Al




_______________________________________________
ippm mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/ippm<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/ippm__;!!BhdT!gDy8HkQIlHamGvp43OZf-RS56h-swktmWW8SG6_ByXuWF4-z511HIzo4YYOTNFcPRYQpcg$>