[113attendees] Network Responsiveness at IETF meeting

[Stuart Cheshire <[e-mail]>]

On Monday morning in IPPM (IP Performance Measurement) Christoph Paasch, Randall Meyer (Apple) and William Hawkins (University of Cincinnati) presented a new testing methodology for testing “Responsiveness under Working Conditions”.

(Full disclosure: I am not unbiased here; I have also been involved with this work.)

For those who haven’t heard about this already, instead of recording the best-case ping time on an idle network (like so many network tests do) the “Responsiveness under Working Conditions” test measures the end-to-end application-layer round trip time while the network is also being used to transfer a realistic amount of data.

In this context, “a realistic amount of data” means enough to saturate typical residential upstream and downstream links, such that the intelligence of the buffer management at the respective bottleneck links is exposed. (Saturating upstream and downstream links is not that unusual -- a single smartphone should be able to do that easily. If you send a photograph or a video to a friend, and your smartphone is *not* sending that as fast as your upstream connection can carry it, then send a bug report to the vendor. Unless you have multi-gigabit Internet service to your home, your smartphone or computer should be easily capable of fully using your network capacity.)

Also, the round trip time is not measured using ICMP “ping” messages, because that risks encouraging vendors to implement special-case optimizations for ICMP packets, which might artificially improve benchmarks numbers but would not actually improve the quality of user experience in any useful way. This test measures the application-layer round trip time because that is what affects user experience. (Every time you are watching a streaming video, and you skip ahead to a different part of that video, or to a different video entirely, and you’re then watching a spinning “buffering” indicator before the new video starts playing, that’s mostly because the new video data you want is stuck at the back of a queue somewhere in the network, waiting behind the previously-requested video data that you no longer want. Avoiding excessive queue buildup in the network avoids the long wait for new data to arrive.)

Finally, rather than reporting round trip time in milliseconds, which is a fairly abstract unit to many people, the test instead reports “round-trips per minute” (RPM). This has the nice property of being a metric where bigger is better (like upload and download throughput) which normal people can relate to. It has a typical range from a few hundred to a few thousand RPM, expressed as a three- or four-digit integer, with no fractions or decimal places required.

While here at the IETF meeting in Vienna, I decided to test the various Wi-Fi networks we have available, and report the Responsiveness scores for each.

Presented without comment, here are the results I measured this afternoon while sitting in tcpm in Grand Klimt Hall 2:

ietf            229 RPM
ietf-nat64      225 RPM
ietf-hotel       30 RPM
Hilton Honors   731 RPM

You can try this test for yourself, either here at the IETF meeting, or on your own home or company network.

On macOS 12 (“Monterey”) you can run the networkQuality command-line tool in a Terminal window to see the network’s RPM score.

On iOS 15 you can enable the RPM test by following the instructions here:
<https://support.apple.com/en-us/HT212313>

On other platforms you can use William Hawkins’s Network Responsiveness test written in Go, available here:
<https://github.com/network-quality/goresponsiveness>

Note that these tools are still under development. One area we are still working on is improving the consistency and repeatability of the results. Ideally repeated runs of the test should give comparable results within ±10%; we are not there yet.

The Internet Draft describing how the responsiveness test works is here:
<https://datatracker.ietf.org/doc/html/draft-ietf-ippm-responsiveness>

There’s a video from the Apple developer conference last year which explains this:
<https://developer.apple.com/videos/play/wwdc2021/10239/>

Monday’s IPPM presentation is here:
<https://datatracker.ietf.org/meeting/113/materials/slides-113-ippm-responsiveness-under-working-conditions-00>

Stuart Cheshire