From nobody Mon Oct 24 13:58:38 2022 Return-Path: X-Original-To: ippm@ietfa.amsl.com Delivered-To: ippm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 51A20C1524C8; Mon, 24 Oct 2022 13:58:35 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.655 X-Spam-Level: X-Spam-Status: No, score=-1.655 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=no autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3tO0SEyfmeP7; Mon, 24 Oct 2022 13:58:31 -0700 (PDT) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5A441C1524CC; Mon, 24 Oct 2022 13:58:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1666645073; bh=iVu4glvXi+i5cQmf+ciW0E1L9U5qyE9U65PoSgbHYf0=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=UhZ0aWCXFGmzCiF131Jh7R4NzNNpfc8gS2S+JPNNL+DRyG8so3pT1lz7lzj5Bm5tZ KhpxWTbrt1zQhslsELSnOSLVsZPWjSOQ4YYEtkh6adeZT55Z2kqDnmKfKA2wnOL0vw xD3GPdyxP7wjYaTQUUBuEEZ7qBe27lNoAfX2in4A= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Received: from smtpclient.apple ([84.157.42.192]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MiacH-1pGL7m1Ufv-00fhn2; Mon, 24 Oct 2022 22:57:53 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) From: Sebastian Moeller In-Reply-To: <26F033E5-2490-414E-8DFF-4ACD27B74075@apple.com> Date: Mon, 24 Oct 2022 22:57:49 +0200 Cc: Glenn Fishbine , Rpm , tsvwg IETF list , IETF IPPM WG , Dave Taht via Starlink , Measurement Analysis and Tools Working Group , discuss Content-Transfer-Encoding: quoted-printable Message-Id: References: <339AB8BC-9628-40E2-9339-77FCFA74488D@gmx.de> <26F033E5-2490-414E-8DFF-4ACD27B74075@apple.com> To: Christoph Paasch X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Provags-ID: V03:K1:DQ7nieuKBGQAbmzqXuiqu8F2qx3Ihl6yWaDPXL7Wy14PgWd2PCB k1g2N93aGoR3vD2gtFF9J0YdNCygRXEssgdVeItwhkPJO4tEstc7rjQDrFcs11KXwopXWjP Mmse4LXQFpHK1e7F8bsMjLjO4t02KeMXr93p48edFMCU40BSfrUbxU+gli+J3ob+s6M2Tng +3GlcRkKo8cSD4FmTpCAQ== X-UI-Out-Filterresults: notjunk:1;V03:K0:xObx3O+klOs=:jDN4VO0zJAnQyvvI0YNe0t pnAxNHUwiay3Ki6A8HjNkoHYoA5K3jt/l3RO8TeV/vK2ezf2lQ0bbhN4RnLsiDPcsp9YPfMSW xQxMZVUFwS9ck/lEMJO+gny8LeR7e+BEyCkygl9UQsd2MmGz/DdgdXheMbcbGmdG8zYhO00dn ie48MWIjiYSzNDiLJ8wl9sx6730WC00u0LFOt5Sex8uP4VQv6ntq2X8k4LbMKpr1X1+37a05n Oc61pJz1YbwH9ug1oUETnckEzigWtBB4o67yuDb3hqUEcdnkZJwmld1LVg7y3mJ8F0NdHqPKm j8y5f+3Fb436AYhrT+Hd73hsbzDOOyjjGAih9nJUv4CHU5+20jNvhYiarhJX/Eh3Nouz5DiaW Z0m8dbyOREMTXNMGJfYQQ45VLSAMLs7jfkGGtf4ZOaRaVybuBavrQZFawAPcZyTkihUfM2i45 7o/Uft3o16y1R0B2hhlMk4KzN6fS5dFaKTFSG9gp9A0DbM+xcj1BXPivRsr45vm/7mSR2T2su P665FBtexZHoM4k9LrSN4ZJHl8SHsU0T3IpJbp/l7lguEOhCjvt6ZEB7UlP2WzhWnqrU+gfOZ 0Fs9RJICmtaiATH8IPkHU061XNc1ti/Ob2O9EXFAj6OANykkXMB2GKVlFehTaSb567pcOJ58I jxBKORG6NjNX4I+xTYIu1QnziHoYOjcC8AjY5T7BRqBEmXzifYKOO0ivF8vhIsU92ePQwN9Cs 3qAku67j9TOquoiBAnwcOrSVc5zTXZCV/K+8d2xJ3XY7LgucpWYXiZg75vp1JJ2gT/VpUZHtx zNWdj0CwQFuNwYIeK/hoeLcDRjINl8zV1rwolovRh5Vjj3YkbCPweI8j/5ZCmXYMG65AUExyl mgjEfm1yQy1esjXH06P173Ke06g2K1DeYfiAPcaaOLq8HV6E1Wn74qKnRrhIdqMSCb6dBwh1s UZFQBijRyhA8UwPiKqb6tSpgcKJIXT+Z/ThbXG5PHUhssB8VDQbSIU1Fry6P2ZRsPMqExL3K6 KlmOGcO78AweNNLiz+1LwVEXjlGUDytt5lGr5OcqzDPvwb5H1eut7bXE/5nIJKwV3XsMoS+5B NyAX9iE0C82HkoCN1CVloZNjyhYlqzKknbYfQL7fmwAZj1LpF99iGOPGIXfWcYSJHj8rKjpQM NmmRWowhDdCyvFc0fQiGi0RSVOxQQ/Pg9ZVULLxO2N0FZsLe4ZRTcUAuSWRd4taHp/Dn36zca 9GCmDewyD21CvxYzUo8Zak/5jN4YTOVRfzhZn50haLh45TPHY7192Iu4UTTEKRmxlyPQoMrhT FQJFM89Ha2TjEOT/xIQrO++uz1gWm+R1Rj6wg4h3C+pzdRIgH4TnBeAsNP7xDwppmJ50YYXxZ CVHH+KSQAvEL3COl19hEw/XqiWOX/Zusk6mDa05AnDLjDtrjYwSW+FXDW/ebo8vANipFlMjX0 qad74ImlZo4v3FAt25GEmx3W2Dz9ZXpKiiACL5rCiHR+2mC3BK7Liwldop8ODjc7GLEGlh6o8 bl2Tf8IeyKmUiGAvd/7/leuzBzn6J1uNoIoxBNNpP4tvT Archived-At: Subject: Re: [ippm] [Starlink] [Rpm] [M-Lab-Discuss] misery metrics & consequences X-BeenThere: ippm@ietf.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: IETF IP Performance Metrics Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2022 20:58:35 -0000 Hi Christoph > On Oct 24, 2022, at 22:08, Christoph Paasch wrote: >=20 > Hello Sebastian, >=20 >> On Oct 23, 2022, at 4:57 AM, Sebastian Moeller via Starlink = wrote: >>=20 >> Hi Glenn, >>=20 >>=20 >>> On Oct 23, 2022, at 02:17, Glenn Fishbine via Rpm = wrote: >>>=20 >>> As a classic died in the wool empiricist, granted that you can = identify "misery" factors, given a population of 1,000 users, how do you = propose deriving a misery index for that population? >>>=20 >>> We can measure download, upload, ping, jitter pretty much without = user intervention. For the measurements you hypothesize, how you you = automatically extract those indecies without subjective user = contamination. >>>=20 >>> I.e. my download speed sucks. Measure the download speed. >>>=20 >>> My isp doesn't fix my problem. Measure what? How? >>>=20 >>> Human survey technology is 70+ years old and it still has problems = figuring out how to correlate opinion with fact. >>>=20 >>> Without an objective measurement scheme that doesn't require human = interaction, the misery index is a cool hypothesis with no way to link = to actual data. What objective measurements can be made? Answer that = and the index becomes useful. Otherwise it's just consumer whining. >>>=20 >>> Not trying to be combative here, in fact I like the concept you = support, but I'm hard pressed to see how the concept can lead to data, = and the data lead to policy proposals. >>=20 >> [SM] So it seems that outside of seemingly simple to test = throughput numbers*, the next most important quality number (or the most = important depending on subjective ranking) is how does latency change = under "load". Absolute latency is also important albeit static high = latency can be worked around within limits so the change under load = seems more relevant.=20 >> All of flent's RRUL test, apple's networkQuality/RPM, and = iperf2's bounceback test offer methods to asses latency change under = load**, as do waveforms bufferbloat tests and even to a degree Ookla's = speedtest.net. IMHO something like latency increase under load or = apple's responsiveness measure RPM (basically the inverse of the latency = under load calculated on a per minute basis, so it scales in the typical = higher numbers are better way, unlike raw latency under load numbers = where smaller is better). >> IMHO what networkQuality is missing ATM is to measure and report = the unloaded RPM as well as the loaded the first gives a measure over = the static latency the second over how well things keep working if = capacity gets tight. They report the base RTT which can be converted to = RPM. As an example: >>=20 >> macbook:~ user$ networkQuality -v >> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 >> Upload capacity: 24.341 Mbps >> Download capacity: 91.951 Mbps >> Upload flows: 20 >> Download flows: 16 >> Responsiveness: High (2123 RPM) >> Base RTT: 16 >> Start: 10/23/22, 13:44:39 >> End: 10/23/22, 13:44:53 >> OS Version: Version 12.6 (Build 21G115) >=20 > You should update to latest macOS: >=20 > $ networkQuality > =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D > Uplink capacity: 326.789 Mbps > Downlink capacity: 446.359 Mbps > Responsiveness: High (2195 RPM) > Idle Latency: 5.833 milli-seconds >=20 > ;-) >=20 [SM] I wish... just updated to the latest and greatest for this = hardware (A1398): macbook-pro:DPZ smoeller$ networkQuality =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 Upload capacity: 7.478 Mbps Download capacity: 2.415 Mbps Upload flows: 16 Download flows: 20 Responsiveness: Low (90 RPM) macbook-pro:DPZ smoeller$ networkQuality -v =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 Upload capacity: 5.830 Mbps Download capacity: 6.077 Mbps Upload flows: 12 Download flows: 20 Responsiveness: Low (56 RPM) Base RTT: 134 Start: 10/24/22, 22:47:48 End: 10/24/22, 22:48:09 OS Version: Version 12.6.1 (Build 21G217) macbook-pro:DPZ smoeller$=20 Still, I only see the "Base RTT" with the -v switch and I am not sure = whether that is identical to your "Idle Latency". I guess I need to convince my employer to exchange that macbook = (actually because the battery starts bulging and not because I am behind = with networkQuality versions ;) ) > But, what I read is: You are suggesting that =E2=80=9CIdle Latency=E2=80= =9D should be expressed in RPM as well? Or, Responsiveness expressed in = millisecond ? [SM] Yes, I am fine with either (or both) the idea is to make it = really easy to see whether/how much "working conditions" deteriorate the = responsiveness / increase the latency-under-load. At least in verbose = mode it would be sweet if nwtworkQuality could expose that information. Regards Sebastian=20 >=20 >=20 > Christoph >=20 >=20 >=20 >>=20 >> Here RPM 2133 corresponds to 60000/2123 =3D 28.26 ms latency under = load, while the Base RTT of 16ms corresponds to 60000/16 =3D 3750 RPM, = son on this link load reduces the responsiveness by 3750-2123 =3D 1627 = RPM a reduction by 100-100*2123/3750 =3D 43.4%, and that is with = competent AQM and scheduling on the router. >>=20 >> Without competent AQM/shaping I get: >> =3D=3D=3D=3D SUMMARY =3D=3D=3D=3D = =20 >> Upload capacity: 15.101 Mbps >> Download capacity: 97.664 Mbps >> Upload flows: 20 >> Download flows: 12 >> Responsiveness: Medium (427 RPM) >> Base RTT: 16 >> Start: 10/23/22, 13:51:50 >> End: 10/23/22, 13:52:06 >> OS Version: Version 12.6 (Build 21G115) >> latency under load: 60000/427 =3D 140.52 ms=20 >> base RPM: 60000/16 =3D 3750 RPM >> reduction RPM: 100-100*427/3750 =3D 88.6% >>=20 >>=20 >> I understand apple's desire to have a single reported number with a = single qualifier medium/high/... because in the end a link is only = reliably usable if responsiveness under load stays acceptable, but with = two numbers it is easier to see what one's ISP could do to help. (I = guess some ISPs might already be unhappy with the single number, so this = needs some diplomacy/tact) >>=20 >> Regards >> Sebastian >>=20 >>=20 >>=20 >> *) Seemingly as quite some ISPs operate their own speedtest servers = in their network and ignore customers not reaching the contracted rates = into speedtest-servers located in different ASs. As the product is = called internet access I a inclined to expect that my ISP maintains = sufficient peering/transit capacity to reach the next tier of AS at my = contracted rate (the EU legislative seems to agree, see EU directive = 2015/2120). >>=20 >> **) Most do by creating load themselves and measuring throughput at = the same time, bounceback IIUC will focus on the latency measurement and = leave the load generation optional (so offers a mode to measure = responsiveness of a live network with minimal measurement traffic). = @Bob, please correct me if this is wrong. >>=20 >>=20 >>>=20 >>>=20 >>> On Fri, Oct 21, 2022, 5:20 PM Dave Taht wrote: >>> One of the best talks I've ever seen on how to measure customer >>> satisfaction properly just went up after the P99 Conference. >>>=20 >>> It's called Misery Metrics. >>>=20 >>> After going through a deep dive as to why and how we think and act = on >>> percentiles, bins, and other statistical methods as to how we use = the >>> web and internet are *so wrong* (well worth watching and thinking >>> about if you are relying on or creating network metrics today), it >>> then points to the real metrics that matter to users and the = ultimate >>> success of an internet business: Timeouts, retries, misses, failed >>> queries, angry phone calls, abandoned shopping carts and loss of >>> engagement. >>>=20 >>> https://www.p99conf.io/session/misery-metrics-consequences/ >>>=20 >>> The ending advice was - don't aim to make a specific percentile >>> acceptable, aim for an acceptable % of misery. >>>=20 >>> I enjoyed the p99 conference more than any conference I've attended = in years. >>>=20 >>> --=20 >>> This song goes out to all the folk that thought Stadia would work: >>> = https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665= 607352320-FXtz >>> Dave T=C3=A4ht CEO, TekLibre, LLC >>>=20 >>> --=20 >>> You received this message because you are subscribed to the Google = Groups "discuss" group. >>> To unsubscribe from this group and stop receiving emails from it, = send an email to discuss+unsubscribe@measurementlab.net. >>> To view this discussion on the web visit = https://groups.google.com/a/measurementlab.net/d/msgid/discuss/CAA93jw4w27= a1EO_QQG7NNkih%2BC3QQde5%3D_7OqGeS9xy9nB6wkg%40mail.gmail.com. >>> _______________________________________________ >>> Rpm mailing list >>> Rpm@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/rpm >>=20 >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink