Re: [tsvwg] links to Canary methods for roll-out of new transport features

["Holland, Jake" <[e-mail]>]

Hi Martin,

From: Martin Duke <[e-mail]>
> Date: Fri,2021-08-13 at 10:14 AM
> The "control" part of the canary experiment is how you identify collateral damage. The L4S hypotheses, informally, are that  
> (1) L4S flows will experience lower latency, etc
> (2) Not-ECT flows, and 3168 flows that traverse most queueing configurations, will not suffer significant degradation.
> (3) 3168 flows will only suffer starvation in queue configurations that are not deployed at scale in the internet.
>
> So the proper design of a canary deployment, IMO, is in two steps:
> (a) Have servers choose between Not-ECT and ECT(0) to establish a baseline.
> (b) servers choose between Not-ECT, ECT(0), and ECT(1) and compare the results.

I don't think I understand this suggestion.

I’m not sure if you’re saying when TCP Prague is turned on, the
server would endeavor to actively send simultaneous competing
non-L4S traffic on secondary flows to the same receiver so they
can detect whether there’s impairment in the secondary flows, like
in the “Out of Band Active Detection” case in the Feb 2021
detection paper (https://arxiv.org/pdf/1911.00710.pdf)?  If so,
this is somewhat more complex than most canary systems--from the
sender you don’t generally have a way to open a second connection
correlated with the first (e.g. when a client is making web requests
to a load-balanced service) so it would need a lot of extra cross-
flow coordination features in the server as compared to a usual
canary test.

But I also don’t see how selecting between ECT(0) and Not-ECT when
establishing a baseline is helpful here, since Non-ECT flows will
get starved the same as ECT(0) flows if they’re traversing a shared
marking classic queue in competition with TCP Prague traffic,
whereas the suggestion makes it seem like the selection between
ECT(0) and N-ECT is meant to do something helpful toward noticing
problems.  (Possible source of confusion: by "3168 flows" in 3), did
you mean ECT(0) flows, thinking only those would be impacted?  Or did
you mean classic traffic in general across a 3168 marking queue?)

This leads me to believe that either I don’t understand the intended
suggestion, or that it didn’t incorporate some of the necessary
considerations of the complicating factors discussed earlier in the
thread.

While the “self-harm” case should in principle be possible to notice
when you do manage to get simultaneous flows for your Prague and
classic traffic across the same marking bottleneck, the “being harmed
by others” case would actually be worse than invisible to the canary,
in that it would corrupt the baseline data and tend to make even the
self-harm detection cases look like noise.  If you had some legit
problem paths with shared marking queues, you’d have results that
look like “yes ECT(0) and N-ECT get impaired for this receiver when
competing with our Prague flows, but they also get impaired randomly
even when we’re not sending competing Prague traffic” (for instance,
when someone else is sending Prague traffic or unresponsive traffic).

For these reasons (and also the reasons Jonathan outlined in response
to Gorry's question about the difference between this and other CC-
related work), I don't see how the canary strategy can be used
effectively here, so to me the detection strategies are a better fit
to the problem (though they do have some of their own challenges, as
previously discussed).

With that said, canary testing would still of course be useful to
detect implementation flaws where L4s/Prague flows systematically
suffer unexpected impairment and I expect would be used as a matter
of course, I'm just saying it would not help with the safety concerns
we've been discussing to detect harm to other people's traffic, as I
don't think they'd be able to see the collateral damage they're
causing.

Best regards,
Jake