Re: [tsvwg] links to Canary methods for roll-out of new transport features

[Martin Duke <[e-mail]>]

Hi Jake,

I don't think I was clear enough.

The assumption in my proposal is that in a large deployment these things
will average out. So there is no intent to create additional connections
solely to detect self-harm. As large deployments will likely have multiple
connections to each endpoint and even larger pools of connections through
important bottlenecks, I assert that if a provider

(1) establishes a Not-ECT and ECT(0) baseline;
(2) Randomly turns on ECT(1) for 1/3 (?) of flows; and
(3) Both Not-ECT and ECT(0) meet some standard of
non-degradation/non-starvation in the aggregate.

then that is strong evidence that the hypothesis that the problematic 3168
queueing is not operating on the internet.

As for including both Not-ECT and ECT(0) in the baseline, I seem to recall
someone making separate claims about the impact of L4S on each; if that's
inaccurate, I have no objection to collapsing the control into one or the
other.

I hope that's clearer.

Meanwhile, I am not crystal-clear as to whether the proponents intend to
deploy L4S endpoints with Prague, something else that meets the Prague
requirements, or nothing at all.

Warmly,
Martin

On Sat, Aug 21, 2021 at 3:39 PM Holland, Jake <[e-mail]> wrote:

> Hi Martin,
>
> From: Martin Duke <[e-mail]>
> > Date: Fri,2021-08-13 at 10:14 AM
> > The "control" part of the canary experiment is how you identify
> collateral damage. The L4S hypotheses, informally, are that
> > (1) L4S flows will experience lower latency, etc
> > (2) Not-ECT flows, and 3168 flows that traverse most queueing
> configurations, will not suffer significant degradation.
> > (3) 3168 flows will only suffer starvation in queue configurations that
> are not deployed at scale in the internet.
> >
> > So the proper design of a canary deployment, IMO, is in two steps:
> > (a) Have servers choose between Not-ECT and ECT(0) to establish a
> baseline.
> > (b) servers choose between Not-ECT, ECT(0), and ECT(1) and compare the
> results.
>
> I don't think I understand this suggestion.
>
> I’m not sure if you’re saying when TCP Prague is turned on, the
> server would endeavor to actively send simultaneous competing
> non-L4S traffic on secondary flows to the same receiver so they
> can detect whether there’s impairment in the secondary flows, like
> in the “Out of Band Active Detection” case in the Feb 2021
> detection paper (https://arxiv.org/pdf/1911.00710.pdf)?  If so,
> this is somewhat more complex than most canary systems--from the
> sender you don’t generally have a way to open a second connection
> correlated with the first (e.g. when a client is making web requests
> to a load-balanced service) so it would need a lot of extra cross-
> flow coordination features in the server as compared to a usual
> canary test.
>
> But I also don’t see how selecting between ECT(0) and Not-ECT when
> establishing a baseline is helpful here, since Non-ECT flows will
> get starved the same as ECT(0) flows if they’re traversing a shared
> marking classic queue in competition with TCP Prague traffic,
> whereas the suggestion makes it seem like the selection between
> ECT(0) and N-ECT is meant to do something helpful toward noticing
> problems.  (Possible source of confusion: by "3168 flows" in 3), did
> you mean ECT(0) flows, thinking only those would be impacted?  Or did
> you mean classic traffic in general across a 3168 marking queue?)
>
> This leads me to believe that either I don’t understand the intended
> suggestion, or that it didn’t incorporate some of the necessary
> considerations of the complicating factors discussed earlier in the
> thread.
>
> While the “self-harm” case should in principle be possible to notice
> when you do manage to get simultaneous flows for your Prague and
> classic traffic across the same marking bottleneck, the “being harmed
> by others” case would actually be worse than invisible to the canary,
> in that it would corrupt the baseline data and tend to make even the
> self-harm detection cases look like noise.  If you had some legit
> problem paths with shared marking queues, you’d have results that
> look like “yes ECT(0) and N-ECT get impaired for this receiver when
> competing with our Prague flows, but they also get impaired randomly
> even when we’re not sending competing Prague traffic” (for instance,
> when someone else is sending Prague traffic or unresponsive traffic).
>
> For these reasons (and also the reasons Jonathan outlined in response
> to Gorry's question about the difference between this and other CC-
> related work), I don't see how the canary strategy can be used
> effectively here, so to me the detection strategies are a better fit
> to the problem (though they do have some of their own challenges, as
> previously discussed).
>
> With that said, canary testing would still of course be useful to
> detect implementation flaws where L4s/Prague flows systematically
> suffer unexpected impairment and I expect would be used as a matter
> of course, I'm just saying it would not help with the safety concerns
> we've been discussing to detect harm to other people's traffic, as I
> don't think they'd be able to see the collateral damage they're
> causing.
>
> Best regards,
> Jake
>
>
>