Re: [tsvwg] links to Canary methods for roll-out of new transport features

[Martin Duke <[e-mail]>]

(As an individual)

The "control" part of the canary experiment is how you identify collateral
damage. The L4S hypotheses, informally, are that
(1) L4S flows will experience lower latency, etc
(2) Not-ECT flows, and 3168 flows that traverse most queueing
configurations, will not suffer significant degradation.
(3) 3168 flows will only suffer starvation in queue configurations that are
not deployed at scale in the internet.

So the proper design of a canary deployment, IMO, is in two steps:
(a) Have servers choose between Not-ECT and ECT(0) to establish a baseline.
(b) servers choose between Not-ECT, ECT(0), and ECT(1) and compare the
results.

On Fri, Jul 30, 2021 at 4:24 AM Gorry Fairhurst <[e-mail]>
wrote:

>
> On 30/07/2021 11:21, Jonathan Morton wrote:
> > On Friday, 30 July 2021, Gorry (erg) wrote:
> >> I am not sure though, I think many CC- related topics can have
> potential collateral damage, and we have managed to deploy these gradually
> and improve the transport were necessary. So, what is different here to
> exploring methods such as larger Initial Window, BBR, Hystart, etc.
> > Hystart makes a transport strictly less aggressive, by exiting the
> exponential growth phase early and continuing with much slower linear or
> polynomial growth.  There is no possibility of collateral damage except in
> case of implementation bugs.  This is an excellent use case for canary
> testing, and I would endorse its use with Hystart++.
> >
> > Large IWs do have potential for collateral damage, but the conditions
> that would trigger it (small buffers) result in effects (high loss to IW)
> that are easily noticed by the transport employing it.  This is therefore
> also suitable for canary testing.
> >
> > BBR is not so clear a case.  I am pleased that Google is taking a
> relatively cautious approach to deploying it in an Internet facing context,
> has designed it with standards-track CC coexistence in mind, and seeks to
> improve it when problems are reported and verified.  However, the ECN
> response introduced with BBRv2 is not standards-track compatible, and the
> likely collateral damage when operating in a shared AQM bottleneck is not
> easily noticed by the BBR transport itself, especially when those
> circumstances arise only infrequently.  Canary testing is thus an
> incomplete solution there.
> >
> > The entire problem here is that L4S is likely to cause *externalised*
> collateral damage which is not easily noticed by the L4S transport itself.
> Unless great care is taken to watch for such problems, canary testing will
> therefore fail to find them.
> >
> > What is more, canary testing has as a prerequisite the confidence of
> correct operation and design gained though lab testing.  Lab testing of L4S
> has not given any of that confidence, thus progression to canary testing
> would be inappropriate.
> >
> >   - Jonathan Morton
>
> I was responding to a request to provide references to canary approaches.
>
> Gorry
>
>