Re: [PANRG] On the new text on ECN: draft-irtf-panrg-what-not-to-do-16

[Bob Briscoe <[e-mail]>]

Spencer,

The apocrypha in the draft needs to tell the whole story. The current 
message is that everything was done right, except we got blind-sided by 
a bug. No.

ECN involved a /three/-part deployment, client, server /and/ bottleneck 
AQM. The crashing home routers pushed back the client part, and servers 
until they finally realized they could enable ECN without making anyone 
more vulnerable to the home router crashes. But my email was about why 
the network part never got deployed (until the more recent FQ-CoDel and 
CAKE deployments).

Even once the bug was out of the way, the deployment pain for networks 
wasn't worth the small performance gain. In the language of the draft, 
ECN didn't "Outperform End-to-end Protocol Mechanisms", which had had 
plenty of time to mask losses in other ways before ECN arrived  (all 
evidenced in my original posting).

The final deployment of the network part in FQ-CoDel and CAKE is more 
difficult to explain. I think that was down to the nature of the open 
source communities that built them. A network operator would have 
assessed whether ECN's performance gain was worth the cost of deployment 
(as in the story in my original posting). But for the open source 
community the ability to function efficiently was enough. It wasn't 
driven by the more traditional cost-benefit analysis approach of 
commercial operators.

In contrast, if you look at most machines at the broadband access 
bottleneck from the major vendors (Nokia, Ericsson, Siemens, etc), they 
didn't implement AQM, let alone ECN, 'cos the operators they were 
selling to tendered for QoS features, which they understood as something 
the network alone provides, not something it helps end-systems to 
provide for themselves (as in AQM & ECN).


Bob

On 11/03/2021 23:37, Spencer Dawkins at IETF wrote:
> Thanks for continuing this discussion on the mailing list. If I might 
> add this ...
>
> On Thu, Mar 11, 2021 at 11:13 AM Bob Briscoe <[e-mail] 
> <mailto:[e-mail]>> wrote:
>
>     Gorry, and panrg,
>
>     I'd agree with Gorry, about one chance per generation.
>
>
> My understanding (and we're talking about this to understand it 
> better) is that there are two things in play with the early ECN 
> deployment experiment.
>
> Yes, routers crashing when they say non-zero ECN bits was in a device 
> which (at the time) wasn't likely to be updated or replaced for a 
> number of years (see "equipment generation"), but the second point 
> (and I think it was Kireeti who said this well) was that once you've 
> updated or replaced all of the problematic equipment, the people who 
> turned ECN off didn't immediately turn ECN back on (I remember the 
> phrase "a bad taste in their mouths" mentioned multiple times.
>
> So, as Kireeti and others observed, generations between updates are 
> getting shorter, but it's not clear that the ability of people to 
> forgive and forget past experiences is getting shorter in the same 
> way. I'm remembering the quote from Oscar Wilde, "Second marriage is 
> the triumph of hope over experience" - I think we're talking about the 
> same thing as "second marriage".
>
> Best,
>
> Spencer
>
>     I'd also say that there are two main prongs to the ECN story. Not
>     just
>     the "one chance" point.
>     The other is "Outperforming End-to-end Protocol Mechanisms"
>
>     When I was asked to write the business case for deploying ECN in
>     BT, my
>     colleagues in the tech strategy team pushed me on this point of
>     outperforming e2e mechanisms. Jamal Salim's performance evaluation
>     [RFC2884] showed ECN gave no performance benefit, except for short
>     flows. I had to admit that e2e FEC for short flows would be more
>     likely
>     to win out. And at the time, I'd just found Damon Wischik paper about
>     how little the extra percentage of traffic volume would be, if every
>     sender just duplicated the first few packets of each flow [Wischik08].
>
>     So the business case flat-lined. End systems had long since worked
>     out
>     tonnes of loss-hiding tricks. There was far too great a risk that, by
>     the time the 3-part deployment had got anywhere (sender, receiver and
>     network), the problem would have been solved e2e, and the high-risk
>     high-cost investment would all have been wasted.
>
>     This was actually the start of my journey to realize that the "ECN =
>     drop" rule was the problem, 'cos it disallowed the real benefit of
>     ECN -
>     to cut queuing delay, by providing a finer-grained signal than
>     loss. I
>     did some calculations to work out that the noise in the delay
>     signal was
>     too great to get queue delay down as low as would be achievable
>     with ECN
>     (which could use virtual queues once ISPs realized that bandwidth had
>     become plentiful enough). That was actually when I started working on
>     chirping (bringing in Mirja as a research fellow) and came to the
>     conclusion that we could only get a better delay signal out of the
>     noise
>     by creating more noise with the chirps. That's when I realized the
>     chirps should only be used at start-up, and ECN was the only way
>     to keep
>     queueing extremely low under load. Today you can see the limits of
>     using
>     e2e delay to reduce queuing in BBR, although of course BBR is
>     unlikely
>     to be the last word in e2e delay reduction.
>
>     I know it's unlikely that every ISP went through such a rigorous
>     exercise, but still none deployed it - probably intuitively
>     reaching the
>     same conclusion. The main reason being the deployment pain wasn't
>     worth
>     the small performance gain. Which is a TL;DR summary of all the above.
>
>     The issue with Linksys home routers crashing wasn't a biggie in that
>     assessment of ECN - there were few enough of that model around by
>     then
>     that it could be worked round with pre-deployment validation testing
>     from the OS. This supports Gorry's "one chance per generation" point.
>
>     Reference
>     [Wischik08] Wischik, D. "Short Messages" Philosophical
>     Transactions of
>     the Royal Society A, 2008, 366, 1941-1953
>
>
>     Bob
>
>     On 11/03/2021 16:17, Gorry Fairhurst wrote:
>     > Resend: On 11/03/2021 15:06, Gorry Fairhurst wrote:
>     >> HI,
>     >>
>     >> Two observations on ECN in what may have been learned?
>     >>
>     >> * I think the "one chance" is per one /Generation/ of equipment
>     for
>     >> hardware, possibly less time for software updates - but hard to
>     >> eliminate deployed critical bugs completely.
>     >>
>     >> * "Measure widely before, but important also gently test during
>     >> initial deployment", to avoid the pain when there are issues and
>     >> therefore to provide a chance to refine the design if there is a
>     >> problem, and reduce the pain of trying.
>     >>
>     >> I like the additional thought from Eric (in the RG meeting) on
>     >> mitigating user pain by using fallback techniques, so they do not
>     >> share your pain when something happens to go wong.
>     >>
>     >> ————
>     >>
>     >> Is this a typo?: /Cannot be recovered at TCP layer/
>     >> - it seems like a partial sentence, does need a /This.../
>     >>
>     >> Gorry
>     >>
>     >> _______________________________________________
>     >> Panrg mailing list
>     >> [e-mail] <mailto:[e-mail]>
>     >> https://www.irtf.org/mailman/listinfo/panrg
>     <https://www.irtf.org/mailman/listinfo/panrg>
>     >
>     >
>     > _______________________________________________
>     > Panrg mailing list
>     > [e-mail] <mailto:[e-mail]>
>     > https://www.irtf.org/mailman/listinfo/panrg
>     <https://www.irtf.org/mailman/listinfo/panrg>
>
>     -- 
>     ________________________________________________________________
>     Bob Briscoe http://bobbriscoe.net/ <http://bobbriscoe.net/>
>
>     _______________________________________________
>     Panrg mailing list
>     [e-mail] <mailto:[e-mail]>
>     https://www.irtf.org/mailman/listinfo/panrg
>     <https://www.irtf.org/mailman/listinfo/panrg>
>
>
> _______________________________________________
> Panrg mailing list
> [e-mail]
> https://www.irtf.org/mailman/listinfo/panrg

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/