Re: [PANRG] On the new text on ECN: draft-irtf-panrg-what-not-to-do-16

[Gorry Fairhurst <[e-mail]>]

This is helpful ... yesterday I was refelecting on the benefits of using 
AQM/RED (a different story) ... however, I'd agree with Jake: to me 
successful deployment of features that require new configuration needs 
operators to perceive the achievable benefits for the bulk of the traffic.

Gorry

On 12/03/2021 07:24, Holland, Jake wrote:
>
> Hi panrg,
>
> I think I agree with Bob’s main point, if I’ve understood it correctly.
>
> A note worth highlighting for the innocent ECN bystanders here:
>
> Most of the benefits from FQ-codel and CAKE come from running an AQM 
> and making it into a throughput bottleneck, which (I think 
> uncontroversially) seems to have given substantial latency benefits to 
> most of the people who’ve deployed it.
>
> The ECN marking itself (as opposed to the AQM that’s doing the 
> marking, but could just as easily drop the packet instead of marking) 
> usually provides only the marginal benefit of avoiding one 
> retransmitted packet, usually on a TCP fast retransmit (plus a few 
> benefits from corner cases where you wouldn’t have gotten 3 dup acks, 
> or a lost packet otherwise would have hurt more than usual).
>
> There’s a fundamental problem that derives from treating loss as the 
> right congestion signal to use, and then defining the CE marking to 
> mean “respond the same as loss”.  In some sense the big problem 
> underneath this came from landing on Reno instead of Vegas (along with 
> policing aggressive senders) [1].
>
> (It might even be right to say that almost all our later 
> congestion-related problems derive from that early “Reno vs. Vegas” 
> outcome, the fallout from which still hasn’t really been fixed yet.  
> This for instance is a key reason that large tail-drop buffers 
> consistently cause poor latency, and I can state with pretty good 
> confidence that it dramatically complicated the deployment of 
> alternative congestion controllers.)
>
> Anyway, this point about ECN performance is well-taken IMO, and it 
> makes me think maybe there’s a good addition to the “test gently 
> during initial deployment” advice that’s something like “characterize 
> and quantify the achievable benefits for the bulk of the traffic 
> that’s supposed to get benefits before nailing down semantics for 
> scarce codepoints” [2].
>
> Best regards,
>
> Jake
>
> PS:  I’m not vouching for Bob’s side comments about characterizing the 
> reasoning behind the ISP and open-source communities’ decision 
> processes; I imagine those a little differently but just as 
> speculatively, so I won’t debate them here.  I’m just agreeing with 
> the specific point about the realizable performance gains from 3168 
> ECN as compared to what’s achievable without using up the codepoints.
>
> [1] I’ll just give one reference to probably my favorite explanation 
> of the dynamics here, for those who haven’t read up on this:
>
> “Analysis and improvement of fairness between TCP Reno and Vegas for 
> deployment of TCP Vegas to the Internet”,**Hasegawa et.al.
>
> https://ieeexplore.ieee.org/abstract/document/896302
>
> [2] Worth noting is that on Sally Floyd’s ECN page, referenced from 
> RFC 3168, the first question in the list of open ECN issues is about 
> quantifying the benefits of ECN for TCP:
>
> http://www.icir.org/floyd/ecn.html#issues
>
>
>       “Open issues for ECN:
>
>
>       -What are the quantitative benefits of ECN for *TCP*?”
>
> *From: *Bob Briscoe <[e-mail]>
> *Date: *Thursday, March 11, 2021 at 4:41 PM
> *To: *Spencer Dawkins at IETF <[e-mail]>
> *Cc: *"[e-mail]" <[e-mail]>
> *Subject: *Re: [PANRG] On the new text on ECN: 
> draft-irtf-panrg-what-not-to-do-16
>
> Spencer,
>
> The apocrypha in the draft needs to tell the whole story. The current 
> message is that everything was done right, except we got blind-sided 
> by a bug. No.
>
> ECN involved a /three/-part deployment, client, server /and/ 
> bottleneck AQM. The crashing home routers pushed back the client part, 
> and servers until they finally realized they could enable ECN without 
> making anyone more vulnerable to the home router crashes. But my email 
> was about why the network part never got deployed (until the more 
> recent FQ-CoDel and CAKE deployments).
>
> Even once the bug was out of the way, the deployment pain for networks 
> wasn't worth the small performance gain. In the language of the draft, 
> ECN didn't "Outperform End-to-end Protocol Mechanisms", which had had 
> plenty of time to mask losses in other ways before ECN arrived  (all 
> evidenced in my original posting).
>
> The final deployment of the network part in FQ-CoDel and CAKE is more 
> difficult to explain. I think that was down to the nature of the open 
> source communities that built them. A network operator would have 
> assessed whether ECN's performance gain was worth the cost of 
> deployment (as in the story in my original posting). But for the open 
> source community the ability to function efficiently was enough. It 
> wasn't driven by the more traditional cost-benefit analysis approach 
> of commercial operators.
>
> In contrast, if you look at most machines at the broadband access 
> bottleneck from the major vendors (Nokia, Ericsson, Siemens, etc), 
> they didn't implement AQM, let alone ECN, 'cos the operators they were 
> selling to tendered for QoS features, which they understood as 
> something the network alone provides, not something it helps 
> end-systems to provide for themselves (as in AQM & ECN).
>
>
> Bob
>
> On 11/03/2021 23:37, Spencer Dawkins at IETF wrote:
>
>     Thanks for continuing this discussion on the mailing list. If I
>     might add this ...
>
>     On Thu, Mar 11, 2021 at 11:13 AM Bob Briscoe <[e-mail]
>     <mailto:[e-mail]>> wrote:
>
>         Gorry, and panrg,
>
>         I'd agree with Gorry, about one chance per generation.
>
>     My understanding (and we're talking about this to understand it
>     better) is that there are two things in play with the early ECN
>     deployment experiment.
>
>     Yes, routers crashing when they say non-zero ECN bits was in a
>     device which (at the time) wasn't likely to be updated or replaced
>     for a number of years (see "equipment generation"), but the second
>     point (and I think it was Kireeti who said this well) was that
>     once you've updated or replaced all of the problematic equipment,
>     the people who turned ECN off didn't immediately turn ECN back on
>     (I remember the phrase "a bad taste in their mouths" mentioned
>     multiple times.
>
>     So, as Kireeti and others observed, generations between updates
>     are getting shorter, but it's not clear that the ability of people
>     to forgive and forget past experiences is getting shorter in the
>     same way. I'm remembering the quote from Oscar Wilde, "Second
>     marriage is the triumph of hope over experience" - I think we're
>     talking about the same thing as "second marriage".
>
>     Best,
>
>     Spencer
>
>         I'd also say that there are two main prongs to the ECN story.
>         Not just
>         the "one chance" point.
>         The other is "Outperforming End-to-end Protocol Mechanisms"
>
>         When I was asked to write the business case for deploying ECN
>         in BT, my
>         colleagues in the tech strategy team pushed me on this point of
>         outperforming e2e mechanisms. Jamal Salim's performance
>         evaluation
>         [RFC2884] showed ECN gave no performance benefit, except for
>         short
>         flows. I had to admit that e2e FEC for short flows would be
>         more likely
>         to win out. And at the time, I'd just found Damon Wischik
>         paper about
>         how little the extra percentage of traffic volume would be, if
>         every
>         sender just duplicated the first few packets of each flow
>         [Wischik08].
>
>         So the business case flat-lined. End systems had long since
>         worked out
>         tonnes of loss-hiding tricks. There was far too great a risk
>         that, by
>         the time the 3-part deployment had got anywhere (sender,
>         receiver and
>         network), the problem would have been solved e2e, and the
>         high-risk
>         high-cost investment would all have been wasted.
>
>         This was actually the start of my journey to realize that the
>         "ECN =
>         drop" rule was the problem, 'cos it disallowed the real
>         benefit of ECN -
>         to cut queuing delay, by providing a finer-grained signal than
>         loss. I
>         did some calculations to work out that the noise in the delay
>         signal was
>         too great to get queue delay down as low as would be
>         achievable with ECN
>         (which could use virtual queues once ISPs realized that
>         bandwidth had
>         become plentiful enough). That was actually when I started
>         working on
>         chirping (bringing in Mirja as a research fellow) and came to the
>         conclusion that we could only get a better delay signal out of
>         the noise
>         by creating more noise with the chirps. That's when I realized
>         the
>         chirps should only be used at start-up, and ECN was the only
>         way to keep
>         queueing extremely low under load. Today you can see the
>         limits of using
>         e2e delay to reduce queuing in BBR, although of course BBR is
>         unlikely
>         to be the last word in e2e delay reduction.
>
>         I know it's unlikely that every ISP went through such a rigorous
>         exercise, but still none deployed it - probably intuitively
>         reaching the
>         same conclusion. The main reason being the deployment pain
>         wasn't worth
>         the small performance gain. Which is a TL;DR summary of all
>         the above.
>
>         The issue with Linksys home routers crashing wasn't a biggie
>         in that
>         assessment of ECN - there were few enough of that model around
>         by then
>         that it could be worked round with pre-deployment validation
>         testing
>         from the OS. This supports Gorry's "one chance per generation"
>         point.
>
>         Reference
>         [Wischik08] Wischik, D. "Short Messages" Philosophical
>         Transactions of
>         the Royal Society A, 2008, 366, 1941-1953
>
>
>         Bob
>
>         On 11/03/2021 16:17, Gorry Fairhurst wrote:
>         > Resend: On 11/03/2021 15:06, Gorry Fairhurst wrote:
>         >> HI,
>         >>
>         >> Two observations on ECN in what may have been learned?
>         >>
>         >> * I think the "one chance" is per one /Generation/ of
>         equipment for
>         >> hardware, possibly less time for software updates - but
>         hard to
>         >> eliminate deployed critical bugs completely.
>         >>
>         >> * "Measure widely before, but important also gently test
>         during
>         >> initial deployment", to avoid the pain when there are
>         issues and
>         >> therefore to provide a chance to refine the design if there
>         is a
>         >> problem, and reduce the pain of trying.
>         >>
>         >> I like the additional thought from Eric (in the RG meeting) on
>         >> mitigating user pain by using fallback techniques, so they
>         do not
>         >> share your pain when something happens to go wong.
>         >>
>         >> ————
>         >>
>         >> Is this a typo?: /Cannot be recovered at TCP layer/
>         >> - it seems like a partial sentence, does need a /This.../
>         >>
>         >> Gorry
>         >>
>         >> _______________________________________________
>         >> Panrg mailing list
>         >> [e-mail] <mailto:[e-mail]>
>         >> https://www.irtf.org/mailman/listinfo/panrg
>         <https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>         >
>         >
>         > _______________________________________________
>         > Panrg mailing list
>         > [e-mail] <mailto:[e-mail]>
>         > https://www.irtf.org/mailman/listinfo/panrg
>         <https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>
>         -- 
>         ________________________________________________________________
>         Bob Briscoe http://bobbriscoe.net/
>         <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPep-Eu0I$>
>
>         _______________________________________________
>         Panrg mailing list
>         [e-mail] <mailto:[e-mail]>
>         https://www.irtf.org/mailman/listinfo/panrg
>         <https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>
>
>
>     _______________________________________________
>
>     Panrg mailing list
>
>     [e-mail]  <mailto:[e-mail]>
>
>     https://www.irtf.org/mailman/listinfo/panrg  <https://urldefense.com/v3/__https:/www.irtf.org/mailman/listinfo/panrg__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPdIRbtu0$>
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/  <https://urldefense.com/v3/__http:/bobbriscoe.net/__;!!GjvTz_vk!FxUkNSuED9rm82ew4jMi4N0VnQsoyAwFhzcCQSB4d7zUi8K7RgWu0biPep-Eu0I$>
>
> _______________________________________________
> Panrg mailing list
> [e-mail]
> https://www.irtf.org/mailman/listinfo/panrg