Re: [ippm] Preliminary measurement comparison of "Working Latency" metrics

Hello!

   Speaking for networkQuality, I might be able to add some color here.  The capacity values output from the tool are actually goodput and not raw throughput that I think Ookla and UDPST are showing. If you’re wanting to compare download-only goodput or RPM values, I would also suggest running networkQuality with -s (along with -v) to get a closer apples to apples comparison to Ookla or your own udpst. “networkQuality -s” will run upload and download tests sequentially. 

In a subsequent message, you mentioned having to use Apple’s servers to use networkQuality or Will’s goresponsiveness. If you want to test on your own infrastructure,  we do provide some sample code on github. I recommend the go-server for ease of deployment/hacking on (https://github.com/network-quality/server/tree/main/go) The default default networkQuality servers we run should be reasonably located to you in Chicago.

As for the low RPM scores, it very well could be hardware. I’ve been comparing a couple of DOCSIS 3.1 modems with 1.2 Gbps provisioned, and one definitely performs worse than the other when doing simultaneous upload/download tests. 

-rm

> On Oct 24, 2022, at 3:35 PM, MORTON JR., AL <acmorton@att.com> wrote:
> 
> Hi RPM friends and IPPM'ers,
> 
> I was wondering what a comparison of some of the "working latency" metrics would look like, so I ran some tests using a service on DOCSIS 3.1, with the downlink provisioned for 1Gbps.
> 
> I intended to run apple's networkQuality, UDPST (RFC9097), and Ookla Speedtest with as similar connectivity as possible (but we know that the traffic will diverge to different servers and we can't change that aspect).
> 
> Here's a quick summary of yesterday's results:
> 
> Working Latency & Capacity Summary
> 
> Net Qual                UDPST                        Ookla
> DnCap     RPM           DnCap    RTTmin   RTTVarRnge DnCap    Ping(no load)
> 878       62            970      28       0-19       941      6
> 891       92            970      27       0-20       940      7
> 891       120           966      28       0-22       937      9
> 890       112           970      28       0-21       940      8
> 903       70            970      28       0-16       935      9
> 
> Note: all RPM values were categorized as Low.
> 
> networkQuality downstream capacities are always on the low side compared to others. We would expect about 940Mbps for TCP, and that's mostly what Ookla achieved. I think that a longer test duration might be needed to achieve the actual 1Gbps capacity with networkQuality; intermediate values observed were certainly headed in the right direction. (I recently upgraded to Monterey 12.6 on my MacBook, so should have the latest version.)
> 
> Also, as Sebastian Moeller's message to the list reminded me, I should have run the tests with the -v option to help with comparisons. I'll repeat this test when I can make time.
> 
> The UDPST measurements of RTTmin (minimum RTT observed during the test) and the range of variation above the minimum (RTTVarRnge) add-up to very reasonable responsiveness IMO, so I'm not clear why RPM graded this access and path as "Low". The UDPST server I'm using is in NJ, and I'm in Chicago conducting tests, so the minimum 28ms is typical. UDPST measurements were run on an Ubuntu VM in my MacBook.
> 
> The big disappointment was that the Ookla desktop app I updated over the weekend did not include the new responsiveness metric! I included the ping results anyway, and it was clearly using a server in the nearby area.
> 
> So, I have some more work to do, but I hope this is interesting-enough to start some comparison discussions, and bring-out some suggestions.
> 
> happy testing all,
> Al
> 
> 
> 
> 
> _______________________________________________
> ippm mailing list
> ippm@ietf.org
> https://www.ietf.org/mailman/listinfo/ippm