From nobody Mon Oct 31 21:21:59 2022 Return-Path: X-Original-To: ippm@ietfa.amsl.com Delivered-To: ippm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B6B09C14CF10 for ; Mon, 31 Oct 2022 21:21:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.105 X-Spam-Level: X-Spam-Status: No, score=-2.105 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fXPZj5mI4ZZZ for ; Mon, 31 Oct 2022 21:21:54 -0700 (PDT) Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D3661C14F737 for ; Mon, 31 Oct 2022 21:21:53 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id o4so18602254wrq.6 for ; Mon, 31 Oct 2022 21:21:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=MdMw2yj3FQnPck2we5WQrVZkS3sxUVbqdkAnXI86ol8=; b=IfGu/VLTvG87EzjdoFzmkBdB2NIm2I9zbstfS7rmFRUwzA0MqWgxaItscqljZQM8AE HxrlhIwBSJjYVkW41zNFAbpMMLzXvZSiHYrfzRwoYSnojNU8QZh+f2whjRZBSZnixLYr 3lz2vv+yVhcrqzXfKjaiJgPTvNjqJ3f56BBgSH/ky4GCFpQmmFE5sJswZWFkEZ5km3ee WqhyK2zh7SdQyDQ5kRyNxp3VWdwRIHAXP/S+9pOFTQ38MD5vLS2Tpcklf3M9Qwot10vP X64ia69R3S3NBCMXmC3EDyoPhoagmYa6dYgkN/QDwycKiIOkQ7/DnSGhpnJfqmunN0S3 fQXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MdMw2yj3FQnPck2we5WQrVZkS3sxUVbqdkAnXI86ol8=; b=uB+g8Glv2ML6jqv+4iDtiptrmbqOipLQmn1qio3qF0XUtLBHVduLDqKB9W3KAciFcU 3QiJkRNA3QhU7aDt/6n4ErNBNsBOMUCyuS8Xn8XwuUBt6syTZCHR4bg1vtS/IKsXcSq/ SV2kyPmvvD0oXaFRtWBVGP/uRyA+ROBup/ucjwzUcVp96iOP6UHye4BC5GI9Lz3WDtEM T/60JHAyBBp7zM79Z+X/KJ+e4FP1uv5gR4EHBmyQiF4f+HryxTaPQL4vi42iBJk1+2ek EhNJnw0dh7vQutdKndUlpek4sdRdmNx2AvLn4NrHrCYiFDNIu6Y+BMUkF4gaeWshVeRx CM4g== X-Gm-Message-State: ACrzQf3gJkKKC3awH/k8B76Pv4yc17yV3M+pKvSUw319wI66etnE0Oqz cegaUshdkbospQezoCHmkvuCY1pEsodmSsR4xJWKBwLpQ0E= X-Google-Smtp-Source: AMsMyM7Cm13C68rhTAlC9/xI83qTEeXM0ICPRPLtjKXAYwiu1MwBVmDTe2ZRfEr6NtNDGTpz6KDwXlhnaqXYwXutvvo= X-Received: by 2002:adf:f352:0:b0:236:ba3a:d58e with SMTP id e18-20020adff352000000b00236ba3ad58emr8131909wrp.430.1667276512087; Mon, 31 Oct 2022 21:21:52 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Mon, 31 Oct 2022 21:21:39 -0700 Message-ID: To: "MORTON JR., AL" Cc: "ippm@ietf.org" , Rpm Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Archived-At: Subject: Re: [ippm] Preliminary measurement comparison of "Working Latency" metrics X-BeenThere: ippm@ietf.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: IETF IP Performance Metrics Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Nov 2022 04:21:58 -0000 Dear Al: OK, I took your udpst tool for a spin. NICE! 120k binary (I STILL work on machines with only 4MB of flash), good feature set, VERY fast, and in very brief testing, seemed to be accurate in the starlink case, though it's hard to tell with them as the rate changes every 15s. I filed a couple bug reports on trivial stuff: https://github.com/BroadbandForum/obudpst/issues/8 (Adding diffserv and ecn washing or marking detection would be a nice feature to have) Aside from the sheer joy coming from the "it compiles! and runs!" phase I haven't looked much further. I left a copy running on one of my starlink testbeds - fremont.starlink.taht.net - if anyone wants to try it. It's instrumented with netperf, flent, irtt, iperf2 (not quite the latest version from bob, but close), and now udpst, and good to about a gbit. nice tool! Has anyone here played with crusader? ( https://github.com/Zoxc/crusader ) On Mon, Oct 31, 2022 at 4:30 PM Dave Taht wrote: > > On Mon, Oct 31, 2022 at 1:41 PM MORTON JR., AL wrote: > > > > have you tried irtt? (https://github.com/heistp/irtt ) > > I have not. Seems like a reasonable tool for UDP testing. The feature I= didn't like in my scan of the documentation is the use of Inter-packet del= ay variation (IPDV) instead of packet delay variation (PDV): variation from= the minimum (or reference) delay. The morbidly curious can find my analysi= s in RFC 5481: https://datatracker.ietf.org/doc/html/rfc5481 > > irtt was meant to simulate high speed voip and one day > videoconferencing. Please inspect the json output > for other metrics. Due to OS limits it is typically only accurate to a > 3ms interval. One thing it does admirably is begin to expose the > sordid sump of L2 behaviors in 4g, 5g, wifi, and other wireless > technologies, as well as request/grant systems like cable and gpon, > especially when otherwise idle. > > Here is a highres plot of starlink's behaviors from last year: > https://forum.openwrt.org/t/cake-w-adaptive-bandwidth-historic/108848/323= 8 > > clearly showing them "optimizing for bandwidth" and changing next sat > hop, and about a 40ms interval of buffering between these switches. > I'd published elsewhere, if anyone cares, a preliminary study of what > starlink's default behaviors did to cubic and BBR... > > > > > irtt's use of IPDV means that the results won=E2=80=99t compare with UD= PST, and possibly networkQuality. But I may give it a try anyway... > > The more the merrier! Someday the "right" metrics will arrive. > > As a side note, this paper focuses on RAN uplink latency > https://dl.ifip.org/db/conf/itc/itc2021/1570740615.pdf which I think > is a major barrier to most forms of 5G actually achieving good > performance in a FPS game, if it is true for more RANs. I'd like more > to be testing uplink latencies idle and with load, on all > technologies. > > > > > thanks again, Dave. > > Al > > > > > -----Original Message----- > > > From: Dave Taht > > > Sent: Monday, October 31, 2022 12:52 PM > > > To: MORTON JR., AL > > > Cc: ippm@ietf.org; Rpm > > > Subject: Re: [ippm] Preliminary measurement comparison of "Working La= tency" > > > metrics > > > > > > Thank you very much for the steer to RFC9097. I'd completely missed t= hat. > > > > > > On Mon, Oct 31, 2022 at 9:04 AM MORTON JR., AL wro= te: > > > > > > > > (astute readers may have guessed that I pressed "send" too soon on = previous > > > message...) > > > > > > > > I also conducted upstream tests this time, here are the results: > > > > (capacity in Mbps, delays in ms, h and m are RPM categories, High a= nd > > > Medium) > > > > > > > > Net Qual UDPST (RFC9097) Ook= la > > > > UpCap RPM DelLD DelMin UpCap RTTmin RTTrange UpC= ap > > > Ping(no load) > > > > 34 1821 h 33ms 11ms 23 (42) 28 0-252 22 = 8 > > > > 22 281 m 214ms 8ms 27 (52) 25 5-248 22 = 8 > > > > 22 290 m 207ms 8ms 27 (55) 28 0-253 22 = 9 > > > > 21 330 m 182ms 11ms 23 (44) 28 0-255 22 = 7 > > > > 22 334 m 180ms 9ms 33 (56) 25 0-255 22 = 9 > > > > > > > > The Upstream capacity measurements reflect an interesting feature t= hat we > > > can reliably and repeatably measure with UDPST. The first ~3 seconds = of > > > upstream data experience a "turbo mode" of ~50Mbps. UDPST displays th= is > > > behavior in its 1 second sub-interval measurements and has a bimodal = reporting > > > option that divides the complete measurement interval in two time int= ervals to > > > report an initial (turbo) max capacity and a steady-state max capacit= y for the > > > later intervals. The UDPST capacity results present both measurements= : steady- > > > state first. > > > > > > Certainly we can expect bi-model distributions from many ISPs, as, fo= r > > > one thing, the "speedboost" concept remains popular, except that it's > > > misnamed, as it should be called speed-subtract or speed-lose. Worse, > > > it is often configured "sneakily", in that it doesn't kick in for the > > > typical observed duration of the test, for some, they cut the > > > available bandwidth about 20s in, others, 1 or 5 minutes. > > > > > > One of my biggest issues with the rpm spec so far is that it should, > > > at least, sometimes, run randomly longer than the overly short > > > interval it runs for and the tools also allow for manual override of = length. > > > > > > we caught a lot of tomfoolery with flent's rrul test running by defau= lt for > > > 1m. > > > > > > Also, AQMs on the path can take a while to find the optimal drop or m= ark rate. > > > > > > > > > > > The capacity processing in networkQuality and Ookla appear to repor= t the > > > steady-state result. > > > > > > Ookla used to basically report the last result. Also it's not a good > > > indicator of web traffic behavior at all, watching the curve > > > go up much more slowly in their test on say, fiber 2ms, vs starlink, > > > (40ms).... > > > > > > So adding another mode - how quickly is peak bandwidth actually > > > reached, would be nice. > > > > > > I haven't poked into the current iteration of the goresponsiveness > > > test at all: https://urldefense.com/v3/__https://github.com/network- > > > quality/goresponsiveness__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzf= S4NRMsdvb > > > K2bOqw0uMPbFeJ7PxzxTc48iQFubYTxxmyA$ it > > > would be good to try collecting more statistics and histograms and > > > methods of analyzing the data in that libre-source version. > > > > > > How does networkQuality compare vs a vs your tool vs a vs goresponsiv= eness? > > > > > > >I watched the upstream capacity measurements on the Ookla app, and c= ould > > > easily see the initial rise to 40-50Mbps, then the drop to ~22Mbps fo= r most of > > > the test which determined the final result. > > > > > > I tend to get upset when I see ookla's new test flash a peak result i= n > > > the seconds and then settle on some lower number somehow. > > > So far as I know they are only sampling the latency every 250ms. > > > > > > > > > > > The working latency is about 200ms in networkQuality and about 280m= s as > > > measured by UDPST (RFC9097). Note that the networkQuality minimum del= ay is > > > ~20ms lower than the UDPST RTTmin, so this accounts for some of the d= ifference > > > in working latency. Also, we used the very dynamic Type C load > > > adjustment/search algorithm in UDPST during all of this testing, whic= h could > > > explain the higher working latency to some degree. > > > > > > > > So, it's worth noting that the measurements needed for assessing wo= rking > > > latency/responsiveness are available in the UDPST utility, and that t= he UDPST > > > measurements are conducted on UDP transport (used by a growing fracti= on of > > > Internet traffic). > > > > > > Thx, didn't know of this work til now! > > > > > > have you tried irtt? > > > > > > > > > > > comments welcome of course, > > > > Al > > > > > > > > > -----Original Message----- > > > > > From: ippm On Behalf Of MORTON JR., AL > > > > > Sent: Sunday, October 30, 2022 8:09 PM > > > > > To: ippm@ietf.org > > > > > Subject: Re: [ippm] Preliminary measurement comparison of "Workin= g > > > Latency" > > > > > metrics > > > > > > > > > > > > > > > Hi again RPM friends and IPPM'ers, > > > > > > > > > > As promised, I repeated the tests shared last week, this time usi= ng both > > > the > > > > > verbose (-v) and sequential (-s) dwn/up test options of networkQu= ality. I > > > > > followed Sebastian's calculations as well. > > > > > > > > > > Working Latency & Capacity Summary > > > > > > > > > > Net Qual UDPST O= okla > > > > > DnCap RPM DelLD DelMin DnCap RTTmin RTTrange D= nCap > > > > > Ping(no load) > > > > > 885 916 m 66ms 8ms 970 28 0-20 9= 40 8 > > > > > 888 1355 h 44ms 8ms 966 28 0-23 9= 40 8 > > > > > 891 1109 h 54ms 8ms 968 27 0-19 9= 40 9 > > > > > 887 1141 h 53ms 11ms 966 27 0-18 9= 37 7 > > > > > 884 1151 h 52ms 9ms 968 28 0-20 9= 37 9 > > > > > > > > > > With the sequential test option, I noticed that networkQuality ac= hieved > > > nearly > > > > > the maximum capacity reported almost immediately at the start of = a test. > > > > > However, the reported capacities are low by about 60Mbps, especia= lly when > > > > > compared to the Ookla TCP measurements. > > > > > > > > > > The loaded delay (DelLD) is similar to the UDPST RTTmin + (the hi= gh end of > > > the > > > > > RTTrange), for example 54ms compared to (27+19=3D46). Most of the > > > networkQuality > > > > > RPM measurements were categorized as "High". There doesn't seem t= o be much > > > > > buffering in the downstream direction. > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: ippm On Behalf Of MORTON JR., AL > > > > > > Sent: Monday, October 24, 2022 6:36 PM > > > > > > To: ippm@ietf.org > > > > > > Subject: [ippm] Preliminary measurement comparison of "Working = Latency" > > > > > > metrics > > > > > > > > > > > > > > > > > > Hi RPM friends and IPPM'ers, > > > > > > > > > > > > I was wondering what a comparison of some of the "working laten= cy" > > > metrics > > > > > > would look like, so I ran some tests using a service on DOCSIS = 3.1, with > > > the > > > > > > downlink provisioned for 1Gbps. > > > > > > > > > > > > I intended to run apple's networkQuality, UDPST (RFC9097), and = Ookla > > > > > Speedtest > > > > > > with as similar connectivity as possible (but we know that the = traffic > > > will > > > > > > diverge to different servers and we can't change that aspect). > > > > > > > > > > > > Here's a quick summary of yesterday's results: > > > > > > > > > > > > Working Latency & Capacity Summary > > > > > > > > > > > > Net Qual UDPST Ookla > > > > > > DnCap RPM DnCap RTTmin RTTVarRnge DnCap P= ing(no > > > load) > > > > > > 878 62 970 28 0-19 941 6 > > > > > > 891 92 970 27 0-20 940 7 > > > > > > 891 120 966 28 0-22 937 9 > > > > > > 890 112 970 28 0-21 940 8 > > > > > > 903 70 970 28 0-16 935 9 > > > > > > > > > > > > Note: all RPM values were categorized as Low. > > > > > > > > > > > > networkQuality downstream capacities are always on the low side= compared > > > to > > > > > > others. We would expect about 940Mbps for TCP, and that's mostl= y what > > > Ookla > > > > > > achieved. I think that a longer test duration might be needed t= o achieve > > > the > > > > > > actual 1Gbps capacity with networkQuality; intermediate values = observed > > > were > > > > > > certainly headed in the right direction. (I recently upgraded t= o > > > Monterey > > > > > 12.6 > > > > > > on my MacBook, so should have the latest version.) > > > > > > > > > > > > Also, as Sebastian Moeller's message to the list reminded me, I= should > > > have > > > > > > run the tests with the -v option to help with comparisons. I'll= repeat > > > this > > > > > > test when I can make time. > > > > > > > > > > > > The UDPST measurements of RTTmin (minimum RTT observed during t= he test) > > > and > > > > > > the range of variation above the minimum (RTTVarRnge) add-up to= very > > > > > > reasonable responsiveness IMO, so I'm not clear why RPM graded = this > > > access > > > > > and > > > > > > path as "Low". The UDPST server I'm using is in NJ, and I'm in = Chicago > > > > > > conducting tests, so the minimum 28ms is typical. UDPST measure= ments > > > were > > > > > run > > > > > > on an Ubuntu VM in my MacBook. > > > > > > > > > > > > The big disappointment was that the Ookla desktop app I updated= over the > > > > > > weekend did not include the new responsiveness metric! I includ= ed the > > > ping > > > > > > results anyway, and it was clearly using a server in the nearby= area. > > > > > > > > > > > > So, I have some more work to do, but I hope this is interesting= -enough > > > to > > > > > > start some comparison discussions, and bring-out some suggestio= ns. > > > > > > > > > > > > happy testing all, > > > > > > Al > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > ippm mailing list > > > > > > ippm@ietf.org > > > > > > > > > > > > > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipp= m__;!!Bhd > > > > > > > > > > > > > > T!hd5MvMQw5eiICQbsfoNaZBUS38yP4YIodBvz1kV5VsX_cGIugVnz5iIkNqi6fRfIQzW= ef_xKqg4$ > > > > > > > > > > _______________________________________________ > > > > > ippm mailing list > > > > > ippm@ietf.org > > > > > > > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipp= m__;!!Bhd > > > > > T!g- > > > FsktB_l9MMSGNUge6FXDkL1npaKtKcyDtWLcTZGpCunxNNCcTImH8YjC9eUT262Wd8q1E= Bpiw$ > > > > > > > > _______________________________________________ > > > > ippm mailing list > > > > ippm@ietf.org > > > > > > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipp= m__;!!Bhd > > > T!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPbFeJ7PxzxTc48= iQFub_gMs > > > KXU$ > > > > > > > > > > > > -- > > > This song goes out to all the folk that thought Stadia would work: > > > https://urldefense.com/v3/__https://www.linkedin.com/posts/dtaht_the-= mushroom- > > > song-activity-6981366665607352320- > > > FXtz__;!!BhdT!giGhURYxqguQCyB3NT8rE0vADdzxcQ2eCzfS4NRMsdvbK2bOqw0uMPb= FeJ7PxzxT > > > c48iQFub34zz4iE$ > > > Dave T=C3=A4ht CEO, TekLibre, LLC > > > > -- > This song goes out to all the folk that thought Stadia would work: > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698136666= 5607352320-FXtz > Dave T=C3=A4ht CEO, TekLibre, LLC --=20 This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666656= 07352320-FXtz Dave T=C3=A4ht CEO, TekLibre, LLC